[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:01:13 dist-test-slave-2s4d krb5kdc[14612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:01:13 dist-test-slave-2s4d krb5kdc[14612](info): set up 2 sockets
Sep 01 20:01:13 dist-test-slave-2s4d krb5kdc[14612](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20250901 20:01:15.667475 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.945s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:01:15 dist-test-slave-2s4d krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:01:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:01:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:15.791599 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:39431
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:39431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:15.888645 14628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:15.888859 14628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:15.888943 14628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:15.891976 14628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:01:15.892030 14628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:15.892064 14628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:15.892099 14628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:01:15.892133 14628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:01:15.896039 14628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:39431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:39431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:15.896960 14628 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:15.897715 14628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:15.903110 14633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:15.903087 14636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:15.903208 14634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:15.903292 14628 server_base.cc:1047] running on GCE node
I20250901 20:01:15.903735 14628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:15.904536 14628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:15.905691 14628 hybrid_clock.cc:648] HybridClock initialized: now 1756756875905669 us; error 34 us; skew 500 ppm
Sep 01 20:01:15 dist-test-slave-2s4d krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756875, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:15.908102 14628 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:01:15.909044 14628 webserver.cc:480] Webserver started at http://127.14.67.126:39167/ using document root <none> and password file <none>
I20250901 20:01:15.909523 14628 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:15.909569 14628 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:15.909711 14628 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:15.911301 14628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "395e7d8ba38a43d68d707e5085be4ae6"
format_stamp: "Formatted at 2025-09-01 20:01:15 on dist-test-slave-2s4d"
server_key: "9424f8375342001891812e3002b195a5"
server_key_iv: "50eb2e29e0f704a584d71267f5187a07"
server_key_version: "encryptionkey@0"
I20250901 20:01:15.911726 14628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "395e7d8ba38a43d68d707e5085be4ae6"
format_stamp: "Formatted at 2025-09-01 20:01:15 on dist-test-slave-2s4d"
server_key: "9424f8375342001891812e3002b195a5"
server_key_iv: "50eb2e29e0f704a584d71267f5187a07"
server_key_version: "encryptionkey@0"
I20250901 20:01:15.914803 14628 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:01:15.916699 14643 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:15.917577 14628 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:15.917663 14628 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "395e7d8ba38a43d68d707e5085be4ae6"
format_stamp: "Formatted at 2025-09-01 20:01:15 on dist-test-slave-2s4d"
server_key: "9424f8375342001891812e3002b195a5"
server_key_iv: "50eb2e29e0f704a584d71267f5187a07"
server_key_version: "encryptionkey@0"
I20250901 20:01:15.917733 14628 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:15.936280 14628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:15.942059 14628 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:15.942232 14628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:15.948923 14695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:39431 every 8 connection(s)
I20250901 20:01:15.948930 14628 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:39431
I20250901 20:01:15.949877 14628 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:01:15.952219 14696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:15.956295 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 14628
I20250901 20:01:15.956430 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:01:15.957284 14696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6: Bootstrap starting.
I20250901 20:01:15.957340 14605 external_mini_cluster.cc:1468] Setting key be0ed21d79682a32bbab041a289bbf8f
I20250901 20:01:15.959486 14696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:15.960180 14696 log.cc:826] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:15.961736 14696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6: No bootstrap required, opened a new log
I20250901 20:01:15.963907 14696 raft_consensus.cc:357] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395e7d8ba38a43d68d707e5085be4ae6" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39431 } }
I20250901 20:01:15.964061 14696 raft_consensus.cc:383] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:15.964094 14696 raft_consensus.cc:738] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395e7d8ba38a43d68d707e5085be4ae6, State: Initialized, Role: FOLLOWER
I20250901 20:01:15.964469 14696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395e7d8ba38a43d68d707e5085be4ae6" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39431 } }
I20250901 20:01:15.964586 14696 raft_consensus.cc:397] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:01:15.964653 14696 raft_consensus.cc:491] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:01:15.964762 14696 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:15.965490 14696 raft_consensus.cc:513] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395e7d8ba38a43d68d707e5085be4ae6" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39431 } }
I20250901 20:01:15.965772 14696 leader_election.cc:304] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 395e7d8ba38a43d68d707e5085be4ae6; no voters:
I20250901 20:01:15.966017 14696 leader_election.cc:290] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:01:15.966154 14701 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:15.966383 14701 raft_consensus.cc:695] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 395e7d8ba38a43d68d707e5085be4ae6, State: Running, Role: LEADER
I20250901 20:01:15.966706 14701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395e7d8ba38a43d68d707e5085be4ae6" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39431 } }
I20250901 20:01:15.967097 14696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:01:15.967757 14701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 395e7d8ba38a43d68d707e5085be4ae6. Latest consensus state: current_term: 1 leader_uuid: "395e7d8ba38a43d68d707e5085be4ae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395e7d8ba38a43d68d707e5085be4ae6" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39431 } } }
I20250901 20:01:15.967885 14701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:15.968644 14702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "395e7d8ba38a43d68d707e5085be4ae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395e7d8ba38a43d68d707e5085be4ae6" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39431 } } }
I20250901 20:01:15.968735 14702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6 [sys.catalog]: This master's current role is: LEADER
Sep 01 20:01:15 dist-test-slave-2s4d krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:15.969511 14708 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:01:15.972469 14708 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:01:15.975342 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:15.959315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.14.67.126:39431)
0901 20:01:15.959799 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:15.959810 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:15.960281 (+ 471us) server_negotiation.cc:407] Connection header received
0901 20:01:15.962365 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:15.962380 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:15.962681 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:15.962945 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:15.964591 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:15.965394 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:15.966071 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:15.966381 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:15.970057 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:15.970077 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:15.970088 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:15.970123 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:15.972284 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:15.972878 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:15.972883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:15.972888 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:15.972974 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:15.973285 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:15.973289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:15.973290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:15.973654 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:15.973846 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:15.974471 (+ 625us) server_negotiation.cc:299] Negotiation successful
0901 20:01:15.974723 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":137,"threads_started":1}
I20250901 20:01:15.977559 14708 catalog_manager.cc:1349] Generated new cluster ID: 1f1f10df6e7d4d3dbfb5a0c079b34c18
I20250901 20:01:15.977622 14708 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:01:15.982867 14708 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:01:15.983899 14708 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:01:15.998054 14708 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 395e7d8ba38a43d68d707e5085be4ae6: Generated new TSK 0
I20250901 20:01:15.998670 14708 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:16.049623 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:39431
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:16.149284 14724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:16.149497 14724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:16.149585 14724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:16.152649 14724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:16.152710 14724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:16.152827 14724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:01:16.156842 14724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:39431
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:16.157806 14724 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:16.158619 14724 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:16.164669 14730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:16.164798 14724 server_base.cc:1047] running on GCE node
W20250901 20:01:16.164661 14729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:16.164661 14732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:16.165323 14724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:16.165825 14724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:16.166961 14724 hybrid_clock.cc:648] HybridClock initialized: now 1756756876166953 us; error 26 us; skew 500 ppm
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756876, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:16.169543 14724 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:01:16.170445 14724 webserver.cc:480] Webserver started at http://127.14.67.65:34931/ using document root <none> and password file <none>
I20250901 20:01:16.170974 14724 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:16.171025 14724 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:16.171192 14724 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:16.172787 14724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "cacfb231e45c4c378bcbbdc5a45fd8e7"
format_stamp: "Formatted at 2025-09-01 20:01:16 on dist-test-slave-2s4d"
server_key: "8984a36157be48f3c09a2dd965f0293b"
server_key_iv: "e72e022e00d6a4779cd52d73046051f0"
server_key_version: "encryptionkey@0"
I20250901 20:01:16.173214 14724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "cacfb231e45c4c378bcbbdc5a45fd8e7"
format_stamp: "Formatted at 2025-09-01 20:01:16 on dist-test-slave-2s4d"
server_key: "8984a36157be48f3c09a2dd965f0293b"
server_key_iv: "e72e022e00d6a4779cd52d73046051f0"
server_key_version: "encryptionkey@0"
I20250901 20:01:16.176314 14724 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:01:16.178293 14739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:16.179245 14724 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:16.179353 14724 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "cacfb231e45c4c378bcbbdc5a45fd8e7"
format_stamp: "Formatted at 2025-09-01 20:01:16 on dist-test-slave-2s4d"
server_key: "8984a36157be48f3c09a2dd965f0293b"
server_key_iv: "e72e022e00d6a4779cd52d73046051f0"
server_key_version: "encryptionkey@0"
I20250901 20:01:16.179430 14724 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:16.188572 14724 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:16.190829 14724 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:16.190984 14724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:16.191409 14724 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:16.192128 14724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:16.192173 14724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:16.192226 14724 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:16.192265 14724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:16.200223 14724 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:46257
I20250901 20:01:16.200258 14852 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:46257 every 8 connection(s)
I20250901 20:01:16.201067 14724 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:01:16.204284 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 14724
I20250901 20:01:16.204383 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:01:16.204592 14605 external_mini_cluster.cc:1468] Setting key a3ae894b7d9462d9eab007f34fda0311
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756876, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:16.212872 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:16.202615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:48013 (local address 127.14.67.126:39431)
0901 20:01:16.202735 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:16.202739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:16.203308 (+ 569us) server_negotiation.cc:407] Connection header received
0901 20:01:16.204120 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:16.204124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:16.204194 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:16.204282 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:16.205600 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.206150 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:16.207262 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.207402 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:16.209413 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:16.209436 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:16.209441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:16.209474 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:16.211118 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:16.211574 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:16.211580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:16.211582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:16.211633 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:16.211861 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:16.211864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:16.211867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:16.212049 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:16.212160 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:16.212567 (+ 407us) server_negotiation.cc:299] Negotiation successful
0901 20:01:16.212696 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
I20250901 20:01:16.213274 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:16.202814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39431 (local address 127.14.67.65:48013)
0901 20:01:16.203188 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:16.203220 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:16.203921 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:16.204471 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:16.204482 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:16.204874 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:16.205451 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:16.205463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.206265 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:16.206268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:16.207116 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:16.207129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.207332 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:16.207946 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:16.207968 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:16.209273 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:16.211223 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:16.211227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:16.211252 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:16.211457 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:16.211713 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:16.211715 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:16.211717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:16.211784 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:16.212142 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:16.212146 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:16.212337 (+ 191us) client_negotiation.cc:769] Sending connection context
0901 20:01:16.212496 (+ 159us) client_negotiation.cc:240] Negotiation successful
0901 20:01:16.212678 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":85,"threads_started":1}
I20250901 20:01:16.214299 14853 heartbeater.cc:344] Connected to a master server at 127.14.67.126:39431
I20250901 20:01:16.214531 14853 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:16.215035 14853 heartbeater.cc:507] Master 127.14.67.126:39431 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
I20250901 20:01:16.216368 14659 ts_manager.cc:194] Registered new tserver with Master: cacfb231e45c4c378bcbbdc5a45fd8e7 (127.14.67.65:46257)
I20250901 20:01:16.217805 14659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:48013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:16.245316 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:39431
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:16.344044 14860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:16.344223 14860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:16.344313 14860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:16.347282 14860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:16.347335 14860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:16.347450 14860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:01:16.351840 14860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:39431
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:16.352746 14860 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:16.353473 14860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:16.359181 14865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:16.359200 14866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:16.359359 14860 server_base.cc:1047] running on GCE node
W20250901 20:01:16.359447 14868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:16.359767 14860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:16.360241 14860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:16.361383 14860 hybrid_clock.cc:648] HybridClock initialized: now 1756756876361373 us; error 26 us; skew 500 ppm
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756876, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:16.363874 14860 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:01:16.364792 14860 webserver.cc:480] Webserver started at http://127.14.67.66:45423/ using document root <none> and password file <none>
I20250901 20:01:16.365276 14860 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:16.365324 14860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:16.365504 14860 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:16.367126 14860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "0c9a9a820ce648408faa061f0e80d6da"
format_stamp: "Formatted at 2025-09-01 20:01:16 on dist-test-slave-2s4d"
server_key: "eee89b21c1e168cc2f379bab178c6a29"
server_key_iv: "54eca64977b32811c70e37748a2ea671"
server_key_version: "encryptionkey@0"
I20250901 20:01:16.367583 14860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "0c9a9a820ce648408faa061f0e80d6da"
format_stamp: "Formatted at 2025-09-01 20:01:16 on dist-test-slave-2s4d"
server_key: "eee89b21c1e168cc2f379bab178c6a29"
server_key_iv: "54eca64977b32811c70e37748a2ea671"
server_key_version: "encryptionkey@0"
I20250901 20:01:16.370806 14860 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250901 20:01:16.372771 14875 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:16.373677 14860 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:16.373771 14860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "0c9a9a820ce648408faa061f0e80d6da"
format_stamp: "Formatted at 2025-09-01 20:01:16 on dist-test-slave-2s4d"
server_key: "eee89b21c1e168cc2f379bab178c6a29"
server_key_iv: "54eca64977b32811c70e37748a2ea671"
server_key_version: "encryptionkey@0"
I20250901 20:01:16.373854 14860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:16.383452 14860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:16.386010 14860 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:16.386164 14860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:16.386674 14860 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:16.387403 14860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:16.387457 14860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:16.387495 14860 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:16.387533 14860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:16.396267 14860 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:45489
I20250901 20:01:16.396312 14988 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:45489 every 8 connection(s)
I20250901 20:01:16.397078 14860 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:01:16.399845 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 14860
I20250901 20:01:16.399931 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:01:16.400120 14605 external_mini_cluster.cc:1468] Setting key c4c2b10bebcb42e6051db1813da64003
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756876, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.67.67@KRBTEST.COM; defaulting to no policy
I20250901 20:01:16.408737 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:16.398631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:60085 (local address 127.14.67.126:39431)
0901 20:01:16.398774 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:16.398777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:16.399342 (+ 565us) server_negotiation.cc:407] Connection header received
0901 20:01:16.400163 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:16.400166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:16.400237 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:16.400311 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:16.401606 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.402092 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:16.403370 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.403515 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:16.405697 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:16.405715 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:16.405718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:16.405739 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:16.407069 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:16.407503 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:16.407509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:16.407511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:16.407551 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:16.407806 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:16.407809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:16.407811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:16.407939 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:16.408025 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:16.408477 (+ 452us) server_negotiation.cc:299] Negotiation successful
0901 20:01:16.408588 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250901 20:01:16.409286 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:16.398844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39431 (local address 127.14.67.66:60085)
0901 20:01:16.399212 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:16.399244 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:16.399979 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:16.400465 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:16.400472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:16.400912 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:16.401462 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:16.401483 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.402203 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:16.402207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:16.403248 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:16.403259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.403457 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:16.404005 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:16.404039 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:16.405563 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:16.407179 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:16.407184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:16.407194 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:16.407408 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:16.407638 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:16.407640 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:16.407642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:16.407721 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:16.408039 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:16.408046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:16.408255 (+ 209us) client_negotiation.cc:769] Sending connection context
0901 20:01:16.408422 (+ 167us) client_negotiation.cc:240] Negotiation successful
0901 20:01:16.408648 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.67@KRBTEST.COM" created.
I20250901 20:01:16.410281 14989 heartbeater.cc:344] Connected to a master server at 127.14.67.126:39431
I20250901 20:01:16.410542 14989 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:16.411018 14989 heartbeater.cc:507] Master 127.14.67.126:39431 requested a full tablet report, sending...
I20250901 20:01:16.411890 14659 ts_manager.cc:194] Registered new tserver with Master: 0c9a9a820ce648408faa061f0e80d6da (127.14.67.66:45489)
I20250901 20:01:16.412663 14659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:60085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:16.441527 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:39431
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:16.546053 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:16.546247 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:16.546303 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:16.549342 14996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:16.549401 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:16.549480 14996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:01:16.553473 14996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:39431
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:16.554402 14996 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:16.555245 14996 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:16.561654 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:16.561633 15002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:16.561622 15001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:16.561930 14996 server_base.cc:1047] running on GCE node
I20250901 20:01:16.562251 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:16.562790 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:16.563943 14996 hybrid_clock.cc:648] HybridClock initialized: now 1756756876563933 us; error 30 us; skew 500 ppm
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756876, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:16.566569 14996 init.cc:377] Logged in from keytab as kudu/127.14.67.67@KRBTEST.COM (short username kudu)
I20250901 20:01:16.567513 14996 webserver.cc:480] Webserver started at http://127.14.67.67:43831/ using document root <none> and password file <none>
I20250901 20:01:16.568048 14996 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:16.568101 14996 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:16.568269 14996 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:16.569949 14996 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "0736b5b72faa4e17a8af5d39626d1b96"
format_stamp: "Formatted at 2025-09-01 20:01:16 on dist-test-slave-2s4d"
server_key: "c88b6de2878f1b42ee8db8cc81ecc14d"
server_key_iv: "28e7cb543011c68cfef163a3cc5263f9"
server_key_version: "encryptionkey@0"
I20250901 20:01:16.570401 14996 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "0736b5b72faa4e17a8af5d39626d1b96"
format_stamp: "Formatted at 2025-09-01 20:01:16 on dist-test-slave-2s4d"
server_key: "c88b6de2878f1b42ee8db8cc81ecc14d"
server_key_iv: "28e7cb543011c68cfef163a3cc5263f9"
server_key_version: "encryptionkey@0"
I20250901 20:01:16.573498 14996 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250901 20:01:16.575517 15011 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:16.576402 14996 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:01:16.576503 14996 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "0736b5b72faa4e17a8af5d39626d1b96"
format_stamp: "Formatted at 2025-09-01 20:01:16 on dist-test-slave-2s4d"
server_key: "c88b6de2878f1b42ee8db8cc81ecc14d"
server_key_iv: "28e7cb543011c68cfef163a3cc5263f9"
server_key_version: "encryptionkey@0"
I20250901 20:01:16.576592 14996 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:16.592621 14996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:16.595046 14996 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:16.595214 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:16.595706 14996 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:16.596487 14996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:16.596536 14996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:16.596594 14996 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:16.596637 14996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:16.605515 14996 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:39895
I20250901 20:01:16.605549 15124 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:39895 every 8 connection(s)
I20250901 20:01:16.606398 14996 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756876, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:16.616221 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 14996
I20250901 20:01:16.616323 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:01:16.616550 14605 external_mini_cluster.cc:1468] Setting key e2a147c8ada53168c4a792e6abc6eb67
I20250901 20:01:16.617406 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:16.608037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:49621 (local address 127.14.67.126:39431)
0901 20:01:16.608173 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:16.608178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:16.608774 (+ 596us) server_negotiation.cc:407] Connection header received
0901 20:01:16.609528 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:16.609531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:16.609578 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:16.609644 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:16.610829 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.611317 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:16.612251 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.612403 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:16.614344 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:16.614361 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:16.614363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:16.614392 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:16.615648 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:16.616084 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:16.616087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:16.616088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:16.616123 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:16.616427 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:16.616431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:16.616432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:16.616573 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:16.616687 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:16.617137 (+ 450us) server_negotiation.cc:299] Negotiation successful
0901 20:01:16.617247 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250901 20:01:16.618037 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:16.608250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39431 (local address 127.14.67.67:49621)
0901 20:01:16.608641 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:16.608675 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:16.609351 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:16.609788 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:16.609795 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:16.610158 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:16.610705 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:16.610717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.611405 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:16.611408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:16.612137 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:16.612144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.612303 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:16.612887 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:16.612908 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:16.614221 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:16.615751 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:16.615755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:16.615764 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:16.615975 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:16.616206 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:16.616210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:16.616214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:16.616325 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:16.616687 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:16.616695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:16.616903 (+ 208us) client_negotiation.cc:769] Sending connection context
0901 20:01:16.617084 (+ 181us) client_negotiation.cc:240] Negotiation successful
0901 20:01:16.617328 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":90,"threads_started":1}
I20250901 20:01:16.619089 15125 heartbeater.cc:344] Connected to a master server at 127.14.67.126:39431
I20250901 20:01:16.619297 15125 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:16.619761 15125 heartbeater.cc:507] Master 127.14.67.126:39431 requested a full tablet report, sending...
I20250901 20:01:16.620620 14659 ts_manager.cc:194] Registered new tserver with Master: 0736b5b72faa4e17a8af5d39626d1b96 (127.14.67.67:39895)
I20250901 20:01:16.621132 14659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:49621
I20250901 20:01:16.630523 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.67.65@KRBTEST.COM
I20250901 20:01:16.653846 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:16.645770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.14.67.65:46257)
0901 20:01:16.646070 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:16.646075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:16.646099 (+ 24us) server_negotiation.cc:407] Connection header received
0901 20:01:16.646147 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:16.646152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:16.646311 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:16.646433 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:16.647246 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.647918 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:16.648564 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.648736 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:16.650897 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:16.650921 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:16.650932 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:16.650955 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:16.652481 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:16.652855 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:16.652860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:16.652861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:16.652906 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:16.653160 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:16.653162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:16.653163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:16.653321 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:16.653458 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:16.653583 (+ 125us) server_negotiation.cc:299] Negotiation successful
0901 20:01:16.653691 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":107,"threads_started":1}
W20250901 20:01:16.654700 14767 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54108
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.65@KRBTEST.COM
I20250901 20:01:16.676453 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:16.669614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.14.67.65:46257)
0901 20:01:16.669763 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:16.669766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:16.669811 (+ 45us) server_negotiation.cc:407] Connection header received
0901 20:01:16.669914 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:16.669916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:16.669958 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:16.670024 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:16.670765 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.671208 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:16.671782 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:16.671980 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:16.673885 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:16.673900 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:16.673903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:16.673923 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:16.675182 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:16.675531 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:16.675535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:16.675539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:16.675576 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:16.675775 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:16.675779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:16.675780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:16.675902 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:16.676030 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:16.676200 (+ 170us) server_negotiation.cc:299] Negotiation successful
0901 20:01:16.676302 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20250901 20:01:16.677292 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 14724
I20250901 20:01:16.683423 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 14860
I20250901 20:01:16.690742 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 14996
I20250901 20:01:16.696254 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 14628
2025-09-01T20:01:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3001 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[15139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[15139](info): set up 2 sockets
Sep 01 20:01:16 dist-test-slave-2s4d krb5kdc[15139](info): commencing operation
krb5kdc: starting...
W20250901 20:01:18.679060 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.961s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:01:18 dist-test-slave-2s4d krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:01:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:18.805030 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:43237
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:43237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:18.903355 15155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:18.903568 15155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:18.903656 15155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:18.906688 15155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:01:18.906745 15155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:18.906783 15155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:18.906816 15155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:01:18.906852 15155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:01:18.910764 15155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:43237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:43237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:18.911684 15155 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:18.912467 15155 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:18.917460 15163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:18.917532 15160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:18.917538 15161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:18.917754 15155 server_base.cc:1047] running on GCE node
I20250901 20:01:18.918251 15155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:18.919093 15155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:18.920238 15155 hybrid_clock.cc:648] HybridClock initialized: now 1756756878920226 us; error 31 us; skew 500 ppm
Sep 01 20:01:18 dist-test-slave-2s4d krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756878, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:18.922495 15155 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:01:18.923413 15155 webserver.cc:480] Webserver started at http://127.14.67.126:39713/ using document root <none> and password file <none>
I20250901 20:01:18.923900 15155 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:18.923954 15155 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:18.924118 15155 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:18.925678 15155 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "f61941872aab47738af1b83b3d9023ff"
format_stamp: "Formatted at 2025-09-01 20:01:18 on dist-test-slave-2s4d"
server_key: "0ae98f7dc55060bdfdc182c428a48682"
server_key_iv: "1faa6c557c20be82038230a33105ecb2"
server_key_version: "encryptionkey@0"
I20250901 20:01:18.926112 15155 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "f61941872aab47738af1b83b3d9023ff"
format_stamp: "Formatted at 2025-09-01 20:01:18 on dist-test-slave-2s4d"
server_key: "0ae98f7dc55060bdfdc182c428a48682"
server_key_iv: "1faa6c557c20be82038230a33105ecb2"
server_key_version: "encryptionkey@0"
I20250901 20:01:18.929172 15155 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20250901 20:01:18.931224 15170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:18.932206 15155 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250901 20:01:18.932327 15155 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "f61941872aab47738af1b83b3d9023ff"
format_stamp: "Formatted at 2025-09-01 20:01:18 on dist-test-slave-2s4d"
server_key: "0ae98f7dc55060bdfdc182c428a48682"
server_key_iv: "1faa6c557c20be82038230a33105ecb2"
server_key_version: "encryptionkey@0"
I20250901 20:01:18.932408 15155 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:18.948451 15155 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:18.951121 15155 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:18.951283 15155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:18.957628 15155 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:43237
I20250901 20:01:18.957643 15222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:43237 every 8 connection(s)
I20250901 20:01:18.958580 15155 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:01:18.959673 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 15155
I20250901 20:01:18.959779 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:01:18.959996 14605 external_mini_cluster.cc:1468] Setting key 20c3a557ef7a4a97d7eba8ee028eaca8
I20250901 20:01:18.961745 15223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:18.967270 15223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff: Bootstrap starting.
Sep 01 20:01:18 dist-test-slave-2s4d krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:18.969691 15223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:18.970547 15223 log.cc:826] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:18.972427 15223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff: No bootstrap required, opened a new log
I20250901 20:01:18.972916 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:18.961262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.14.67.126:43237)
0901 20:01:18.961648 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:18.961659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:18.961695 (+ 36us) server_negotiation.cc:407] Connection header received
0901 20:01:18.963039 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:18.963061 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:18.963399 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:18.963731 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:18.964423 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:18.965532 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:18.966090 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:18.966346 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:18.968463 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:18.968503 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:18.968515 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:18.968551 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:18.970527 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:18.970908 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:18.970914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:18.970917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:18.970977 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:18.971206 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:18.971208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:18.971209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:18.971581 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:18.971750 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:18.971955 (+ 205us) server_negotiation.cc:299] Negotiation successful
0901 20:01:18.972183 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20250901 20:01:18.975121 15223 raft_consensus.cc:357] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61941872aab47738af1b83b3d9023ff" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 43237 } }
I20250901 20:01:18.975277 15223 raft_consensus.cc:383] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:18.975318 15223 raft_consensus.cc:738] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61941872aab47738af1b83b3d9023ff, State: Initialized, Role: FOLLOWER
I20250901 20:01:18.975663 15223 consensus_queue.cc:260] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61941872aab47738af1b83b3d9023ff" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 43237 } }
I20250901 20:01:18.975783 15223 raft_consensus.cc:397] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:01:18.975826 15223 raft_consensus.cc:491] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:01:18.975899 15223 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:18.976732 15223 raft_consensus.cc:513] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61941872aab47738af1b83b3d9023ff" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 43237 } }
I20250901 20:01:18.977006 15223 leader_election.cc:304] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f61941872aab47738af1b83b3d9023ff; no voters:
I20250901 20:01:18.977241 15223 leader_election.cc:290] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:01:18.977315 15228 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:18.977523 15228 raft_consensus.cc:695] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [term 1 LEADER]: Becoming Leader. State: Replica: f61941872aab47738af1b83b3d9023ff, State: Running, Role: LEADER
I20250901 20:01:18.977828 15228 consensus_queue.cc:237] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61941872aab47738af1b83b3d9023ff" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 43237 } }
I20250901 20:01:18.978168 15223 sys_catalog.cc:564] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:01:18.979343 15229 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f61941872aab47738af1b83b3d9023ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61941872aab47738af1b83b3d9023ff" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 43237 } } }
I20250901 20:01:18.979485 15229 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:18.979669 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61941872aab47738af1b83b3d9023ff. Latest consensus state: current_term: 1 leader_uuid: "f61941872aab47738af1b83b3d9023ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61941872aab47738af1b83b3d9023ff" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 43237 } } }
I20250901 20:01:18.979746 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:18.980036 15237 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:01:18.982853 15237 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:01:18.987587 15237 catalog_manager.cc:1349] Generated new cluster ID: 8ee0a3beeadd486a8056e68a40ffd1b6
I20250901 20:01:18.987649 15237 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:01:18.994124 15237 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:01:18.995260 15237 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:01:19.004725 15237 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f61941872aab47738af1b83b3d9023ff: Generated new TSK 0
I20250901 20:01:19.005445 15237 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:19.050810 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:43237
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:19.149413 15251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:19.149585 15251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:19.149641 15251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:19.152626 15251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:19.152676 15251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:19.152751 15251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:01:19.156585 15251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:43237
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:19.157387 15251 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:19.158085 15251 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:19.163894 15257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:19.163969 15259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:19.163926 15256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:19.164076 15251 server_base.cc:1047] running on GCE node
I20250901 20:01:19.164362 15251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:19.164842 15251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:19.165985 15251 hybrid_clock.cc:648] HybridClock initialized: now 1756756879165970 us; error 34 us; skew 500 ppm
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:19.168350 15251 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:01:19.169232 15251 webserver.cc:480] Webserver started at http://127.14.67.65:37185/ using document root <none> and password file <none>
I20250901 20:01:19.169709 15251 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:19.169762 15251 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:19.169919 15251 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:19.171432 15251 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "89feb36a01644e73b41d430d076bc5b0"
format_stamp: "Formatted at 2025-09-01 20:01:19 on dist-test-slave-2s4d"
server_key: "7c76ea502831044bf4821d3a8e0cba36"
server_key_iv: "46c3a148cfbcf7e187c5c82a779920cd"
server_key_version: "encryptionkey@0"
I20250901 20:01:19.171864 15251 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "89feb36a01644e73b41d430d076bc5b0"
format_stamp: "Formatted at 2025-09-01 20:01:19 on dist-test-slave-2s4d"
server_key: "7c76ea502831044bf4821d3a8e0cba36"
server_key_iv: "46c3a148cfbcf7e187c5c82a779920cd"
server_key_version: "encryptionkey@0"
I20250901 20:01:19.175341 15251 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250901 20:01:19.178242 15266 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:19.179356 15251 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250901 20:01:19.179471 15251 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "89feb36a01644e73b41d430d076bc5b0"
format_stamp: "Formatted at 2025-09-01 20:01:19 on dist-test-slave-2s4d"
server_key: "7c76ea502831044bf4821d3a8e0cba36"
server_key_iv: "46c3a148cfbcf7e187c5c82a779920cd"
server_key_version: "encryptionkey@0"
I20250901 20:01:19.179574 15251 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:19.201373 15251 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:19.204007 15251 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:19.204152 15251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:19.204593 15251 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:19.205356 15251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:19.205403 15251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:19.205451 15251 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:19.205482 15251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:19.213817 15251 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:46603
I20250901 20:01:19.213846 15379 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:46603 every 8 connection(s)
I20250901 20:01:19.214674 15251 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:01:19.215536 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 15251
I20250901 20:01:19.215644 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:01:19.215904 14605 external_mini_cluster.cc:1468] Setting key 565cc07a021b2e61dea83710a426901c
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
I20250901 20:01:19.227038 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.216333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:46175 (local address 127.14.67.126:43237)
0901 20:01:19.216469 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:19.216473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:19.217089 (+ 616us) server_negotiation.cc:407] Connection header received
0901 20:01:19.217900 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:19.217907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:19.217970 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:19.218058 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:19.219540 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.220361 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.221444 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.221704 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.223826 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:19.223843 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:19.223845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:19.223868 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:19.225299 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.225743 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.225745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.225747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.225786 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.225985 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.225987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.225988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.226107 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:19.226175 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:19.226711 (+ 536us) server_negotiation.cc:299] Negotiation successful
0901 20:01:19.226890 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
I20250901 20:01:19.227607 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.216562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:43237 (local address 127.14.67.65:46175)
0901 20:01:19.216959 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:19.216998 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:19.217714 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:19.218182 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:19.218189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:19.218696 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:19.219380 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.219402 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.220475 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.220478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:19.221297 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.221305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.221452 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.221967 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:19.221988 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:19.223655 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:19.225404 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:19.225412 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:19.225423 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:19.225650 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:19.225856 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:19.225858 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:19.225859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:19.225925 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:19.226176 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:19.226180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:19.226438 (+ 258us) client_negotiation.cc:769] Sending connection context
0901 20:01:19.226646 (+ 208us) client_negotiation.cc:240] Negotiation successful
0901 20:01:19.226898 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":89,"threads_started":1}
I20250901 20:01:19.228739 15380 heartbeater.cc:344] Connected to a master server at 127.14.67.126:43237
I20250901 20:01:19.228984 15380 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:19.229509 15380 heartbeater.cc:507] Master 127.14.67.126:43237 requested a full tablet report, sending...
I20250901 20:01:19.230891 15187 ts_manager.cc:194] Registered new tserver with Master: 89feb36a01644e73b41d430d076bc5b0 (127.14.67.65:46603)
I20250901 20:01:19.232391 15187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:46175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:19.262712 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:43237
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:19.359354 15387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:19.359531 15387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:19.359619 15387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:19.362640 15387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:19.362704 15387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:19.362809 15387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:01:19.366701 15387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:43237
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:19.367553 15387 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:19.368284 15387 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:19.374058 15392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:19.374078 15393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:19.374219 15387 server_base.cc:1047] running on GCE node
W20250901 20:01:19.374078 15395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:19.374706 15387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:19.375169 15387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:19.376312 15387 hybrid_clock.cc:648] HybridClock initialized: now 1756756879376298 us; error 27 us; skew 500 ppm
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:19.378798 15387 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:01:19.379693 15387 webserver.cc:480] Webserver started at http://127.14.67.66:40589/ using document root <none> and password file <none>
I20250901 20:01:19.380148 15387 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:19.380198 15387 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:19.380363 15387 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:19.381917 15387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "ad6aefbdfa844e9fa1fd873304b5273e"
format_stamp: "Formatted at 2025-09-01 20:01:19 on dist-test-slave-2s4d"
server_key: "0dd7f24b89765ab4a6e921583501167c"
server_key_iv: "5083266e19170248de71180b7c87e20a"
server_key_version: "encryptionkey@0"
I20250901 20:01:19.382341 15387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "ad6aefbdfa844e9fa1fd873304b5273e"
format_stamp: "Formatted at 2025-09-01 20:01:19 on dist-test-slave-2s4d"
server_key: "0dd7f24b89765ab4a6e921583501167c"
server_key_iv: "5083266e19170248de71180b7c87e20a"
server_key_version: "encryptionkey@0"
I20250901 20:01:19.385394 15387 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250901 20:01:19.387329 15402 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:19.388187 15387 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:19.388291 15387 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "ad6aefbdfa844e9fa1fd873304b5273e"
format_stamp: "Formatted at 2025-09-01 20:01:19 on dist-test-slave-2s4d"
server_key: "0dd7f24b89765ab4a6e921583501167c"
server_key_iv: "5083266e19170248de71180b7c87e20a"
server_key_version: "encryptionkey@0"
I20250901 20:01:19.388377 15387 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:19.402994 15387 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:19.405357 15387 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:19.405525 15387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:19.405969 15387 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:19.406734 15387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:19.406785 15387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:19.406839 15387 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:19.406867 15387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:19.415135 15387 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:38219
I20250901 20:01:19.415163 15515 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:38219 every 8 connection(s)
I20250901 20:01:19.415916 15387 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:01:19.416707 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 15387
I20250901 20:01:19.416795 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:01:19.417004 14605 external_mini_cluster.cc:1468] Setting key 27fdd861a35c709e8cc30b721f2b3c56
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.67@KRBTEST.COM" created.
I20250901 20:01:19.427393 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.417447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:47595 (local address 127.14.67.126:43237)
0901 20:01:19.417569 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:19.417573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:19.418220 (+ 647us) server_negotiation.cc:407] Connection header received
0901 20:01:19.419083 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:19.419090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:19.419156 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:19.419246 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:19.420365 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.420849 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.421927 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.422114 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.424175 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:19.424192 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:19.424194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:19.424218 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:19.425621 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.426065 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.426069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.426070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.426112 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.426356 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.426359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.426361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.426560 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:19.426650 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:19.427117 (+ 467us) server_negotiation.cc:299] Negotiation successful
0901 20:01:19.427234 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250901 20:01:19.427968 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.417701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:43237 (local address 127.14.67.66:47595)
0901 20:01:19.418091 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:19.418131 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:19.418888 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:19.419375 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:19.419382 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:19.419730 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:19.420212 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.420224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.420965 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.420968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:19.421806 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.421814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.421991 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.422712 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:19.422739 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:19.424036 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:19.425736 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:19.425742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:19.425752 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:19.425967 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:19.426193 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:19.426198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:19.426200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:19.426272 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:19.426680 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:19.426689 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:19.426879 (+ 190us) client_negotiation.cc:769] Sending connection context
0901 20:01:19.427069 (+ 190us) client_negotiation.cc:240] Negotiation successful
0901 20:01:19.427313 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":93,"threads_started":1}
I20250901 20:01:19.429205 15516 heartbeater.cc:344] Connected to a master server at 127.14.67.126:43237
I20250901 20:01:19.429443 15516 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:19.430024 15516 heartbeater.cc:507] Master 127.14.67.126:43237 requested a full tablet report, sending...
I20250901 20:01:19.431098 15187 ts_manager.cc:194] Registered new tserver with Master: ad6aefbdfa844e9fa1fd873304b5273e (127.14.67.66:38219)
I20250901 20:01:19.431632 15187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:47595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:19.458899 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:43237
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:19.556854 15523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:19.557039 15523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:19.557098 15523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:19.560026 15523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:19.560074 15523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:19.560180 15523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:01:19.564020 15523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:43237
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:19.564896 15523 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:19.565624 15523 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:19.571324 15528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:19.571326 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:19.571393 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:19.571627 15523 server_base.cc:1047] running on GCE node
I20250901 20:01:19.571940 15523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:19.572449 15523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:19.573585 15523 hybrid_clock.cc:648] HybridClock initialized: now 1756756879573566 us; error 34 us; skew 500 ppm
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:19.576021 15523 init.cc:377] Logged in from keytab as kudu/127.14.67.67@KRBTEST.COM (short username kudu)
I20250901 20:01:19.576850 15523 webserver.cc:480] Webserver started at http://127.14.67.67:45391/ using document root <none> and password file <none>
I20250901 20:01:19.577338 15523 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:19.577412 15523 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:19.577575 15523 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:19.579173 15523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "92960e536f86460c9e299f90d3376876"
format_stamp: "Formatted at 2025-09-01 20:01:19 on dist-test-slave-2s4d"
server_key: "d4dbe4922fc396decefdc72718bfda83"
server_key_iv: "27f4ee9cf841c270bdf877673423feba"
server_key_version: "encryptionkey@0"
I20250901 20:01:19.579598 15523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "92960e536f86460c9e299f90d3376876"
format_stamp: "Formatted at 2025-09-01 20:01:19 on dist-test-slave-2s4d"
server_key: "d4dbe4922fc396decefdc72718bfda83"
server_key_iv: "27f4ee9cf841c270bdf877673423feba"
server_key_version: "encryptionkey@0"
I20250901 20:01:19.582608 15523 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:01:19.584618 15538 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:19.585502 15523 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:19.585597 15523 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "92960e536f86460c9e299f90d3376876"
format_stamp: "Formatted at 2025-09-01 20:01:19 on dist-test-slave-2s4d"
server_key: "d4dbe4922fc396decefdc72718bfda83"
server_key_iv: "27f4ee9cf841c270bdf877673423feba"
server_key_version: "encryptionkey@0"
I20250901 20:01:19.585676 15523 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:19.600003 15523 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:19.602353 15523 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:19.602562 15523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:19.603029 15523 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:19.603786 15523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:19.603834 15523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:19.603889 15523 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:19.603924 15523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:19.612200 15523 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:43657
I20250901 20:01:19.612253 15651 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:43657 every 8 connection(s)
I20250901 20:01:19.613034 15523 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:01:19.623219 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 15523
I20250901 20:01:19.623309 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:01:19.623483 14605 external_mini_cluster.cc:1468] Setting key fef1ceb805e9bcf4e4d7ed0d3295f0a9
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:19.628973 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.614668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:48645 (local address 127.14.67.126:43237)
0901 20:01:19.614806 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:19.614810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:19.615392 (+ 582us) server_negotiation.cc:407] Connection header received
0901 20:01:19.616155 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:19.616158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:19.616220 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:19.616295 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:19.617481 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.617972 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.619127 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.619314 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.626154 (+ 6840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:19.626167 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:19.626170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:19.626192 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:19.627533 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.627950 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.627953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.627954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.627986 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.628195 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.628196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.628198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.628315 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:19.628379 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:19.628778 (+ 399us) server_negotiation.cc:299] Negotiation successful
0901 20:01:19.628866 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250901 20:01:19.629537 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.614881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:43237 (local address 127.14.67.67:48645)
0901 20:01:19.615264 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:19.615298 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:19.615976 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:19.616459 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:19.616480 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:19.616834 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:19.617325 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.617337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.618082 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.618085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:19.618989 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.618997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.619188 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.624434 (+ 5246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:19.624455 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:19.626029 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:19.627648 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:19.627652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:19.627659 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:19.627874 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:19.628054 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:19.628056 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:19.628058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:19.628130 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:19.628389 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:19.628393 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:19.628578 (+ 185us) client_negotiation.cc:769] Sending connection context
0901 20:01:19.628752 (+ 174us) client_negotiation.cc:240] Negotiation successful
0901 20:01:19.628938 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":97,"threads_started":1}
I20250901 20:01:19.630442 15652 heartbeater.cc:344] Connected to a master server at 127.14.67.126:43237
I20250901 20:01:19.630674 15652 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:19.631111 15652 heartbeater.cc:507] Master 127.14.67.126:43237 requested a full tablet report, sending...
I20250901 20:01:19.631975 15187 ts_manager.cc:194] Registered new tserver with Master: 92960e536f86460c9e299f90d3376876 (127.14.67.67:43657)
I20250901 20:01:19.632573 15187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:48645
I20250901 20:01:19.636852 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250901 20:01:19.644929 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.639380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.14.67.126:43237)
0901 20:01:19.639475 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:19.639478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:19.639600 (+ 122us) server_negotiation.cc:407] Connection header received
0901 20:01:19.639688 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:19.639690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:19.639735 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:19.639799 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:19.640485 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.640925 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.641488 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.641595 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.642559 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:19.642572 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:19.642574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:19.642594 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:19.643868 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.644197 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.644199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.644201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.644232 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.644389 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.644391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.644392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.644520 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:19.644615 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:19.644755 (+ 140us) server_negotiation.cc:299] Negotiation successful
0901 20:01:19.644833 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250901 20:01:19.648388 15187 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 20:01:19.650496 15187 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:01:19.661756 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.657870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.14.67.66:38219)
0901 20:01:19.658226 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:19.658234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:19.658270 (+ 36us) server_negotiation.cc:407] Connection header received
0901 20:01:19.658347 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:19.658353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:19.658541 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:19.658697 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:19.659317 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.660088 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.660900 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.661389 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.661500 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:19.661558 (+ 58us) server_negotiation.cc:299] Negotiation successful
0901 20:01:19.661627 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20250901 20:01:19.661828 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.657679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:38219 (local address 127.0.0.1:45946)
0901 20:01:19.658017 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:19.658052 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:19.658190 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:19.658891 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:19.658896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:19.658935 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:19.659185 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.659192 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.660189 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.660193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:19.660750 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.660756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.660859 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.660901 (+ 42us) client_negotiation.cc:769] Sending connection context
0901 20:01:19.660976 (+ 75us) client_negotiation.cc:240] Negotiation successful
0901 20:01:19.661048 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"spinlock_wait_cycles":2816,"thread_start_us":96,"threads_started":1}
I20250901 20:01:19.661988 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.657411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:46603 (local address 127.0.0.1:43174)
0901 20:01:19.658018 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:19.658052 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:19.658191 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:19.659193 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:19.659196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:19.659217 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:19.659476 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.659483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.660839 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.660843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:19.661646 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.661656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.661754 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.661769 (+ 15us) client_negotiation.cc:769] Sending connection context
0901 20:01:19.661816 (+ 47us) client_negotiation.cc:240] Negotiation successful
0901 20:01:19.661871 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"spinlock_wait_cycles":10496,"thread_start_us":78,"threads_started":1}
I20250901 20:01:19.662158 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.658016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:43657 (local address 127.0.0.1:54048)
0901 20:01:19.658429 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:19.658441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:19.658550 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:19.659211 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:19.659215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:19.659247 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:19.659456 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.659462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.660935 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.660938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:19.661799 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.661811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.661908 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.661928 (+ 20us) client_negotiation.cc:769] Sending connection context
0901 20:01:19.661962 (+ 34us) client_negotiation.cc:240] Negotiation successful
0901 20:01:19.662024 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"spinlock_wait_cycles":768,"thread_start_us":83,"threads_started":1}
I20250901 20:01:19.662844 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.657583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.14.67.65:46603)
0901 20:01:19.658800 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:19.658806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:19.658826 (+ 20us) server_negotiation.cc:407] Connection header received
0901 20:01:19.658898 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:19.658902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:19.659062 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:19.659197 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:19.659600 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.660720 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.662128 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.662581 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.662649 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:19.662695 (+ 46us) server_negotiation.cc:299] Negotiation successful
0901 20:01:19.662749 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1112,"thread_start_us":113,"threads_started":1}
I20250901 20:01:19.663291 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.658384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.14.67.67:43657)
0901 20:01:19.658819 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:19.658824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:19.658840 (+ 16us) server_negotiation.cc:407] Connection header received
0901 20:01:19.658897 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:19.658902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:19.659062 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:19.659573 (+ 511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:19.659687 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.660811 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.662437 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.662935 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.663041 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:19.663090 (+ 49us) server_negotiation.cc:299] Negotiation successful
0901 20:01:19.663156 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":140,"threads_started":1}
I20250901 20:01:19.664597 15450 tablet_service.cc:1468] Processing CreateTablet for tablet c2da4ad65be54df98a88590588a1765b (DEFAULT_TABLE table=test-table [id=b7b0f3ab2c9f46bfad934c1bb3f224cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:19.664872 15314 tablet_service.cc:1468] Processing CreateTablet for tablet c2da4ad65be54df98a88590588a1765b (DEFAULT_TABLE table=test-table [id=b7b0f3ab2c9f46bfad934c1bb3f224cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:19.665246 15586 tablet_service.cc:1468] Processing CreateTablet for tablet c2da4ad65be54df98a88590588a1765b (DEFAULT_TABLE table=test-table [id=b7b0f3ab2c9f46bfad934c1bb3f224cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:19.665517 15450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2da4ad65be54df98a88590588a1765b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:19.665582 15314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2da4ad65be54df98a88590588a1765b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:19.665969 15586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2da4ad65be54df98a88590588a1765b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:19.670832 15669 tablet_bootstrap.cc:492] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0: Bootstrap starting.
I20250901 20:01:19.671728 15670 tablet_bootstrap.cc:492] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876: Bootstrap starting.
I20250901 20:01:19.672847 15669 tablet_bootstrap.cc:654] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:19.672938 15671 tablet_bootstrap.cc:492] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e: Bootstrap starting.
I20250901 20:01:19.673381 15670 tablet_bootstrap.cc:654] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:19.673635 15669 log.cc:826] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:19.674000 15670 log.cc:826] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:19.674822 15671 tablet_bootstrap.cc:654] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:19.675411 15669 tablet_bootstrap.cc:492] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0: No bootstrap required, opened a new log
I20250901 20:01:19.675470 15670 tablet_bootstrap.cc:492] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876: No bootstrap required, opened a new log
I20250901 20:01:19.675515 15671 log.cc:826] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:19.675585 15669 ts_tablet_manager.cc:1397] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250901 20:01:19.675640 15670 ts_tablet_manager.cc:1397] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250901 20:01:19.677023 15671 tablet_bootstrap.cc:492] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e: No bootstrap required, opened a new log
I20250901 20:01:19.677183 15671 ts_tablet_manager.cc:1397] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250901 20:01:19.678100 15670 raft_consensus.cc:357] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89feb36a01644e73b41d430d076bc5b0" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 46603 } } peers { permanent_uuid: "92960e536f86460c9e299f90d3376876" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 43657 } } peers { permanent_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 38219 } }
I20250901 20:01:19.678306 15670 raft_consensus.cc:383] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:19.678371 15670 raft_consensus.cc:738] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92960e536f86460c9e299f90d3376876, State: Initialized, Role: FOLLOWER
I20250901 20:01:19.678328 15669 raft_consensus.cc:357] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89feb36a01644e73b41d430d076bc5b0" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 46603 } } peers { permanent_uuid: "92960e536f86460c9e299f90d3376876" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 43657 } } peers { permanent_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 38219 } }
I20250901 20:01:19.678491 15669 raft_consensus.cc:383] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:19.678872 15669 raft_consensus.cc:738] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89feb36a01644e73b41d430d076bc5b0, State: Initialized, Role: FOLLOWER
I20250901 20:01:19.678817 15670 consensus_queue.cc:260] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89feb36a01644e73b41d430d076bc5b0" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 46603 } } peers { permanent_uuid: "92960e536f86460c9e299f90d3376876" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 43657 } } peers { permanent_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 38219 } }
I20250901 20:01:19.679296 15669 consensus_queue.cc:260] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89feb36a01644e73b41d430d076bc5b0" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 46603 } } peers { permanent_uuid: "92960e536f86460c9e299f90d3376876" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 43657 } } peers { permanent_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 38219 } }
I20250901 20:01:19.679450 15652 heartbeater.cc:499] Master 127.14.67.126:43237 was elected leader, sending a full tablet report...
I20250901 20:01:19.679695 15670 ts_tablet_manager.cc:1428] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250901 20:01:19.680296 15669 ts_tablet_manager.cc:1428] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250901 20:01:19.680189 15671 raft_consensus.cc:357] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89feb36a01644e73b41d430d076bc5b0" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 46603 } } peers { permanent_uuid: "92960e536f86460c9e299f90d3376876" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 43657 } } peers { permanent_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 38219 } }
I20250901 20:01:19.680359 15671 raft_consensus.cc:383] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:19.680410 15671 raft_consensus.cc:738] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6aefbdfa844e9fa1fd873304b5273e, State: Initialized, Role: FOLLOWER
I20250901 20:01:19.680599 15380 heartbeater.cc:499] Master 127.14.67.126:43237 was elected leader, sending a full tablet report...
I20250901 20:01:19.680778 15671 consensus_queue.cc:260] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89feb36a01644e73b41d430d076bc5b0" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 46603 } } peers { permanent_uuid: "92960e536f86460c9e299f90d3376876" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 43657 } } peers { permanent_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 38219 } }
I20250901 20:01:19.681761 15516 heartbeater.cc:499] Master 127.14.67.126:43237 was elected leader, sending a full tablet report...
I20250901 20:01:19.681800 15671 ts_tablet_manager.cc:1428] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20250901 20:01:19.715592 15381 tablet.cc:2378] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 20:01:19.779650 15677 raft_consensus.cc:491] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:01:19.779832 15677 raft_consensus.cc:513] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89feb36a01644e73b41d430d076bc5b0" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 46603 } } peers { permanent_uuid: "92960e536f86460c9e299f90d3376876" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 43657 } } peers { permanent_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 38219 } }
I20250901 20:01:19.780929 15677 leader_election.cc:290] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89feb36a01644e73b41d430d076bc5b0 (127.14.67.65:46603), 92960e536f86460c9e299f90d3376876 (127.14.67.67:43657)
I20250901 20:01:19.783574 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.781120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:46603 (local address 127.14.67.66:37787)
0901 20:01:19.781239 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:19.781254 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:19.781340 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:19.781652 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:19.781656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:19.781672 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:19.781874 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.781879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.782612 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.782623 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:19.783174 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.783181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.783284 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.783306 (+ 22us) client_negotiation.cc:769] Sending connection context
0901 20:01:19.783375 (+ 69us) client_negotiation.cc:240] Negotiation successful
0901 20:01:19.783457 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20250901 20:01:19.784006 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.781210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:37787 (local address 127.14.67.65:46603)
0901 20:01:19.781322 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:19.781328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:19.781347 (+ 19us) server_negotiation.cc:407] Connection header received
0901 20:01:19.781490 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:19.781495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:19.781548 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:19.781630 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:19.781975 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.782465 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.783282 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.783806 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.783847 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:19.783885 (+ 38us) server_negotiation.cc:299] Negotiation successful
0901 20:01:19.783926 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250901 20:01:19.784468 15334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2da4ad65be54df98a88590588a1765b" candidate_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89feb36a01644e73b41d430d076bc5b0" is_pre_election: true
I20250901 20:01:19.784560 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.781366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:43657 (local address 127.14.67.66:32995)
0901 20:01:19.781656 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:19.781667 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:19.781763 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:19.781991 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:19.781994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:19.782010 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:19.782239 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.782244 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.783001 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.783004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:19.783677 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:19.783690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.784291 (+ 601us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.784302 (+ 11us) client_negotiation.cc:769] Sending connection context
0901 20:01:19.784376 (+ 74us) client_negotiation.cc:240] Negotiation successful
0901 20:01:19.784439 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":99,"threads_started":1}
I20250901 20:01:19.784582 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.781470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:32995 (local address 127.14.67.67:43657)
0901 20:01:19.781591 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:19.781595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:19.781732 (+ 137us) server_negotiation.cc:407] Connection header received
0901 20:01:19.781863 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:19.781866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:19.781911 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:19.781975 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:19.782341 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.782880 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.783815 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.784212 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.784242 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:19.784414 (+ 172us) server_negotiation.cc:299] Negotiation successful
0901 20:01:19.784478 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250901 20:01:19.784724 15334 raft_consensus.cc:2466] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6aefbdfa844e9fa1fd873304b5273e in term 0.
I20250901 20:01:19.785122 15404 leader_election.cc:304] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89feb36a01644e73b41d430d076bc5b0, ad6aefbdfa844e9fa1fd873304b5273e; no voters:
I20250901 20:01:19.785329 15677 raft_consensus.cc:2802] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:01:19.785354 15606 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2da4ad65be54df98a88590588a1765b" candidate_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92960e536f86460c9e299f90d3376876" is_pre_election: true
I20250901 20:01:19.785408 15677 raft_consensus.cc:491] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:01:19.785440 15677 raft_consensus.cc:3058] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:19.785565 15606 raft_consensus.cc:2466] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6aefbdfa844e9fa1fd873304b5273e in term 0.
I20250901 20:01:19.786461 15677 raft_consensus.cc:513] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89feb36a01644e73b41d430d076bc5b0" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 46603 } } peers { permanent_uuid: "92960e536f86460c9e299f90d3376876" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 43657 } } peers { permanent_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 38219 } }
I20250901 20:01:19.786792 15677 leader_election.cc:290] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [CANDIDATE]: Term 1 election: Requested vote from peers 89feb36a01644e73b41d430d076bc5b0 (127.14.67.65:46603), 92960e536f86460c9e299f90d3376876 (127.14.67.67:43657)
I20250901 20:01:19.787077 15334 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2da4ad65be54df98a88590588a1765b" candidate_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89feb36a01644e73b41d430d076bc5b0"
I20250901 20:01:19.787137 15606 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2da4ad65be54df98a88590588a1765b" candidate_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92960e536f86460c9e299f90d3376876"
I20250901 20:01:19.787184 15334 raft_consensus.cc:3058] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:19.787246 15606 raft_consensus.cc:3058] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:19.788043 15334 raft_consensus.cc:2466] T c2da4ad65be54df98a88590588a1765b P 89feb36a01644e73b41d430d076bc5b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6aefbdfa844e9fa1fd873304b5273e in term 1.
I20250901 20:01:19.788136 15606 raft_consensus.cc:2466] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6aefbdfa844e9fa1fd873304b5273e in term 1.
I20250901 20:01:19.788308 15404 leader_election.cc:304] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89feb36a01644e73b41d430d076bc5b0, ad6aefbdfa844e9fa1fd873304b5273e; no voters:
I20250901 20:01:19.788482 15677 raft_consensus.cc:2802] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:19.788654 15677 raft_consensus.cc:695] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [term 1 LEADER]: Becoming Leader. State: Replica: ad6aefbdfa844e9fa1fd873304b5273e, State: Running, Role: LEADER
I20250901 20:01:19.788908 15677 consensus_queue.cc:237] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89feb36a01644e73b41d430d076bc5b0" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 46603 } } peers { permanent_uuid: "92960e536f86460c9e299f90d3376876" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 43657 } } peers { permanent_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 38219 } }
I20250901 20:01:19.791605 15186 catalog_manager.cc:5582] T c2da4ad65be54df98a88590588a1765b P ad6aefbdfa844e9fa1fd873304b5273e reported cstate change: term changed from 0 to 1, leader changed from <none> to ad6aefbdfa844e9fa1fd873304b5273e (127.14.67.66). New cstate: current_term: 1 leader_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89feb36a01644e73b41d430d076bc5b0" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 46603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92960e536f86460c9e299f90d3376876" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 43657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad6aefbdfa844e9fa1fd873304b5273e" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 38219 } health_report { overall_health: HEALTHY } } }
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.65@KRBTEST.COM
I20250901 20:01:19.812147 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.804934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.14.67.65:46603)
0901 20:01:19.805061 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:19.805067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:19.805168 (+ 101us) server_negotiation.cc:407] Connection header received
0901 20:01:19.805260 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:19.805263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:19.805297 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:19.805359 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:19.806062 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.806466 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.807170 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.807304 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.809335 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:19.809353 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:19.809362 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:19.809383 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:19.811001 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.811318 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.811322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.811324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.811365 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.811511 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.811512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.811514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.811656 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:19.811789 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:19.811882 (+ 93us) server_negotiation.cc:299] Negotiation successful
0901 20:01:19.811965 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756879, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756879, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.67.65@KRBTEST.COM
I20250901 20:01:19.834474 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.827555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.14.67.65:46603)
0901 20:01:19.827699 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:19.827705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:19.827724 (+ 19us) server_negotiation.cc:407] Connection header received
0901 20:01:19.827865 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:19.827869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:19.827913 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:19.827980 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:19.828722 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.829183 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.829790 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.829997 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.831889 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:19.831906 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:19.831909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:19.831930 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:19.833161 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.833522 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.833525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.833527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.833565 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.833768 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.833772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.833774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.833889 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:19.834003 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:19.834213 (+ 210us) server_negotiation.cc:299] Negotiation successful
0901 20:01:19.834322 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.65@KRBTEST.COM
I20250901 20:01:19.856040 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:19.849274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.14.67.65:46603)
0901 20:01:19.849412 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:19.849415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:19.849499 (+ 84us) server_negotiation.cc:407] Connection header received
0901 20:01:19.849608 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:19.849610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:19.849659 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:19.849725 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:19.850448 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.850930 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:19.851524 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:19.851653 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:19.853562 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:19.853578 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:19.853580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:19.853601 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:19.854966 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.855277 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.855280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.855281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.855316 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:19.855465 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:19.855467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:19.855468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:19.855570 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:19.855655 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:19.855794 (+ 139us) server_negotiation.cc:299] Negotiation successful
0901 20:01:19.855873 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20250901 20:01:19.857260 15294 tablet_service.cc:2912] Rejecting scan request for tablet c2da4ad65be54df98a88590588a1765b: Uninitialized: clean time has not yet been initialized
I20250901 20:01:19.858820 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 15251
W20250901 20:01:19.863835 15653 tablet.cc:2378] T c2da4ad65be54df98a88590588a1765b P 92960e536f86460c9e299f90d3376876: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 20:01:19.864996 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 15387
I20250901 20:01:19.870801 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 15523
I20250901 20:01:19.875903 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 15155
2025-09-01T20:01:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3180 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15693](info): set up 2 sockets
Sep 01 20:01:19 dist-test-slave-2s4d krb5kdc[15693](info): commencing operation
krb5kdc: starting...
W20250901 20:01:21.875268 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.977s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:01:21 dist-test-slave-2s4d krb5kdc[15693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:01:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:01:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:21.998610 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:38487
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:36363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:38487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:22.101060 15709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:22.101285 15709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:22.101382 15709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:22.104489 15709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:01:22.104545 15709 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250901 20:01:22.104578 15709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:22.104619 15709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:22.104660 15709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:01:22.104696 15709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:01:22.108760 15709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:36363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:38487
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:38487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:22.109723 15709 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:22.110525 15709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:22.115694 15714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:22.115737 15717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:22.115706 15715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:22.115890 15709 server_base.cc:1047] running on GCE node
I20250901 20:01:22.116446 15709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:22.117251 15709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:22.118398 15709 hybrid_clock.cc:648] HybridClock initialized: now 1756756882118380 us; error 40 us; skew 500 ppm
Sep 01 20:01:22 dist-test-slave-2s4d krb5kdc[15693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:22.120911 15709 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:01:22.121829 15709 webserver.cc:480] Webserver started at http://127.14.67.126:33951/ using document root <none> and password file <none>
I20250901 20:01:22.122325 15709 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:22.122377 15709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:22.122577 15709 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:22.124207 15709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "911d55fff6d2451e8356122292784949"
format_stamp: "Formatted at 2025-09-01 20:01:22 on dist-test-slave-2s4d"
server_key: "a26166af9426b6fc999be4543b280581"
server_key_iv: "a0c73a0bc72797b4679aa726e34fb0f0"
server_key_version: "encryptionkey@0"
I20250901 20:01:22.124661 15709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "911d55fff6d2451e8356122292784949"
format_stamp: "Formatted at 2025-09-01 20:01:22 on dist-test-slave-2s4d"
server_key: "a26166af9426b6fc999be4543b280581"
server_key_iv: "a0c73a0bc72797b4679aa726e34fb0f0"
server_key_version: "encryptionkey@0"
I20250901 20:01:22.127748 15709 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:01:22.129662 15724 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:22.130582 15709 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250901 20:01:22.130681 15709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "911d55fff6d2451e8356122292784949"
format_stamp: "Formatted at 2025-09-01 20:01:22 on dist-test-slave-2s4d"
server_key: "a26166af9426b6fc999be4543b280581"
server_key_iv: "a0c73a0bc72797b4679aa726e34fb0f0"
server_key_version: "encryptionkey@0"
I20250901 20:01:22.130765 15709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:22.152594 15709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:22.159045 15709 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:22.159210 15709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:22.166697 15709 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:38487
I20250901 20:01:22.166702 15786 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:38487 every 8 connection(s)
I20250901 20:01:22.167613 15709 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:01:22.169914 15787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:22.173494 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 15709
I20250901 20:01:22.173591 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:01:22.173827 14605 external_mini_cluster.cc:1468] Setting key 884b4c85be0c9cd6b3b1ce7e11022fab
I20250901 20:01:22.175432 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949: Bootstrap starting.
I20250901 20:01:22.177372 15787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:22.178041 15787 log.cc:826] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:22.179826 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949: No bootstrap required, opened a new log
Sep 01 20:01:22 dist-test-slave-2s4d krb5kdc[15693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:22.182796 15787 raft_consensus.cc:357] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911d55fff6d2451e8356122292784949" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38487 } }
I20250901 20:01:22.182963 15787 raft_consensus.cc:383] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:22.183007 15787 raft_consensus.cc:738] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 911d55fff6d2451e8356122292784949, State: Initialized, Role: FOLLOWER
I20250901 20:01:22.183386 15787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911d55fff6d2451e8356122292784949" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38487 } }
I20250901 20:01:22.183504 15787 raft_consensus.cc:397] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:01:22.183568 15787 raft_consensus.cc:491] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:01:22.183686 15787 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:22.184684 15787 raft_consensus.cc:513] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911d55fff6d2451e8356122292784949" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38487 } }
I20250901 20:01:22.185002 15787 leader_election.cc:304] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 911d55fff6d2451e8356122292784949; no voters:
I20250901 20:01:22.185339 15787 leader_election.cc:290] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:01:22.185412 15792 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:22.185729 15792 raft_consensus.cc:695] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [term 1 LEADER]: Becoming Leader. State: Replica: 911d55fff6d2451e8356122292784949, State: Running, Role: LEADER
I20250901 20:01:22.185979 15792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911d55fff6d2451e8356122292784949" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38487 } }
I20250901 20:01:22.186277 15787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:01:22.186292 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.174981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.14.67.126:38487)
0901 20:01:22.175368 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:22.175377 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:22.175407 (+ 30us) server_negotiation.cc:407] Connection header received
0901 20:01:22.175907 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:22.175921 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:22.176181 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:22.176466 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:22.177328 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.178065 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.178862 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.179125 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.181432 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:22.181448 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:22.181467 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:22.181496 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:22.183832 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.184274 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.184277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.184281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.184340 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.184569 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.184572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.184573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.184885 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:22.185031 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:22.185320 (+ 289us) server_negotiation.cc:299] Negotiation successful
0901 20:01:22.185579 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":130,"threads_started":1}
I20250901 20:01:22.187021 15792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 911d55fff6d2451e8356122292784949. Latest consensus state: current_term: 1 leader_uuid: "911d55fff6d2451e8356122292784949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911d55fff6d2451e8356122292784949" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38487 } } }
I20250901 20:01:22.187121 15792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:22.187700 15793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "911d55fff6d2451e8356122292784949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911d55fff6d2451e8356122292784949" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38487 } } }
I20250901 20:01:22.187791 15793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:22.188167 15799 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:01:22.190492 15799 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:01:22.195685 15799 catalog_manager.cc:1349] Generated new cluster ID: 3de3471de07549729b44673eda0b5629
I20250901 20:01:22.195751 15799 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:01:22.217521 15799 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:01:22.218451 15799 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:01:22.225193 15799 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949: Generated new TSK 0
I20250901 20:01:22.225822 15799 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:22.278380 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:38487
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:36363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:22.384681 15815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:22.384887 15815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:22.384945 15815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:22.388067 15815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:22.388123 15815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:22.388190 15815 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250901 20:01:22.388269 15815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:01:22.392338 15815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:36363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.67.126:38487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:22.393285 15815 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:22.394076 15815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:22.400146 15820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:22.400102 15821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:22.400102 15823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:22.400326 15815 server_base.cc:1047] running on GCE node
I20250901 20:01:22.400686 15815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:22.401180 15815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:22.402323 15815 hybrid_clock.cc:648] HybridClock initialized: now 1756756882402302 us; error 40 us; skew 500 ppm
Sep 01 20:01:22 dist-test-slave-2s4d krb5kdc[15693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:22.404904 15815 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:01:22.405831 15815 webserver.cc:480] Webserver started at http://127.14.67.65:40419/ using document root <none> and password file <none>
I20250901 20:01:22.406327 15815 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:22.406378 15815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:22.406585 15815 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:22.408179 15815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "3da766224ead4bf59b16b202f3ab0547"
format_stamp: "Formatted at 2025-09-01 20:01:22 on dist-test-slave-2s4d"
server_key: "989d8c8361ebba3abdd8e641430f26a9"
server_key_iv: "6ebdb741e05d56b89b8f4a39e84e8661"
server_key_version: "encryptionkey@0"
I20250901 20:01:22.408623 15815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "3da766224ead4bf59b16b202f3ab0547"
format_stamp: "Formatted at 2025-09-01 20:01:22 on dist-test-slave-2s4d"
server_key: "989d8c8361ebba3abdd8e641430f26a9"
server_key_iv: "6ebdb741e05d56b89b8f4a39e84e8661"
server_key_version: "encryptionkey@0"
I20250901 20:01:22.411700 15815 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:01:22.413684 15830 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:22.414625 15815 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250901 20:01:22.414724 15815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "3da766224ead4bf59b16b202f3ab0547"
format_stamp: "Formatted at 2025-09-01 20:01:22 on dist-test-slave-2s4d"
server_key: "989d8c8361ebba3abdd8e641430f26a9"
server_key_iv: "6ebdb741e05d56b89b8f4a39e84e8661"
server_key_version: "encryptionkey@0"
I20250901 20:01:22.414805 15815 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:22.425356 15815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:22.428015 15815 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:22.428194 15815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:22.429562 15815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:22.429625 15815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:22.429688 15815 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:22.429729 15815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 01 20:01:22 dist-test-slave-2s4d krb5kdc[15693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:22.440858 15815 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:38653
I20250901 20:01:22.440882 15945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:38653 every 8 connection(s)
I20250901 20:01:22.441625 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.430175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:44517 (local address 127.14.67.126:38487)
0901 20:01:22.430289 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:22.430293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:22.431092 (+ 799us) server_negotiation.cc:407] Connection header received
0901 20:01:22.431905 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:22.431909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:22.431969 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:22.432046 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:22.433610 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.434421 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.435426 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.435639 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.438145 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:22.438163 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:22.438165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:22.438189 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:22.439862 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.440299 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.440302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.440303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.440343 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.440616 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.440619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.440621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.440808 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:22.440911 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:22.441341 (+ 430us) server_negotiation.cc:299] Negotiation successful
0901 20:01:22.441458 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250901 20:01:22.442175 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.430532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38487 (local address 127.14.67.65:44517)
0901 20:01:22.430968 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:22.431001 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:22.431721 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:22.432209 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:22.432220 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:22.432744 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:22.433472 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.433485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.434598 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.434602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:22.435264 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.435277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.435528 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.436144 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:22.436185 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:22.438007 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:22.439964 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:22.439969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:22.439980 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:22.440196 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:22.440421 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:22.440427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:22.440429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:22.440504 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:22.440912 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:22.440919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:22.441120 (+ 201us) client_negotiation.cc:769] Sending connection context
0901 20:01:22.441295 (+ 175us) client_negotiation.cc:240] Negotiation successful
0901 20:01:22.441543 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20250901 20:01:22.441692 15815 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:01:22.443468 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 15815
I20250901 20:01:22.443581 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:01:22.443847 14605 external_mini_cluster.cc:1468] Setting key b2b7a6a94bc1901097f2cc6b69250c83
I20250901 20:01:22.444048 15946 heartbeater.cc:344] Connected to a master server at 127.14.67.126:38487
I20250901 20:01:22.444223 15946 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:22.444693 15946 heartbeater.cc:507] Master 127.14.67.126:38487 requested a full tablet report, sending...
I20250901 20:01:22.445920 15741 ts_manager.cc:194] Registered new tserver with Master: 3da766224ead4bf59b16b202f3ab0547 (127.14.67.65:38653)
I20250901 20:01:22.447338 15741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:44517
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
I20250901 20:01:22.453402 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.445375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:37113 (local address 127.14.67.126:38487)
0901 20:01:22.445508 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:22.445512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:22.446876 (+ 1364us) server_negotiation.cc:407] Connection header received
0901 20:01:22.446984 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:22.446987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:22.447026 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:22.447089 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:22.447764 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.448527 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.449445 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.449573 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.450647 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:22.450662 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:22.450664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:22.450685 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:22.452035 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.452498 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.452500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.452501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.452532 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.452750 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.452752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.452753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.452859 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:22.452925 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:22.453209 (+ 284us) server_negotiation.cc:299] Negotiation successful
0901 20:01:22.453296 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250901 20:01:22.453452 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.445313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38487 (local address 127.14.67.65:37113)
0901 20:01:22.446793 (+ 1480us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:22.446805 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:22.446899 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:22.447113 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:22.447115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:22.447329 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:22.447621 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.447632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.448681 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.448686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:22.449315 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.449329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.449475 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.449975 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:22.449996 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:22.450418 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:22.452144 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:22.452150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:22.452153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:22.452393 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:22.452607 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:22.452610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:22.452613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:22.452674 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:22.452942 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:22.452947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:22.453048 (+ 101us) client_negotiation.cc:769] Sending connection context
0901 20:01:22.453165 (+ 117us) client_negotiation.cc:240] Negotiation successful
0901 20:01:22.453305 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:22.485530 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:38487
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:36363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:22.585131 15957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:22.585305 15957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:22.585364 15957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:22.588542 15957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:22.588594 15957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:22.588662 15957 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250901 20:01:22.588726 15957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:01:22.592767 15957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:36363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.67.126:38487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:22.593715 15957 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:22.594473 15957 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:22.600255 15965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:22.600262 15962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:22.600265 15963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:22.600700 15957 server_base.cc:1047] running on GCE node
I20250901 20:01:22.600998 15957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:22.601476 15957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:22.602561 15957 hybrid_clock.cc:648] HybridClock initialized: now 1756756882602549 us; error 25 us; skew 500 ppm
Sep 01 20:01:22 dist-test-slave-2s4d krb5kdc[15693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:22.605057 15957 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:01:22.605996 15957 webserver.cc:480] Webserver started at http://127.14.67.66:37837/ using document root <none> and password file <none>
I20250901 20:01:22.606498 15957 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:22.606586 15957 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:22.606753 15957 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:22.608595 15957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "0ac688c1d743428196b1ba6504dc7ca1"
format_stamp: "Formatted at 2025-09-01 20:01:22 on dist-test-slave-2s4d"
server_key: "6a10c67860d517889c30e693ee2b2366"
server_key_iv: "753f490d26c73cbeb7b25b6b026a1be0"
server_key_version: "encryptionkey@0"
I20250901 20:01:22.609030 15957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "0ac688c1d743428196b1ba6504dc7ca1"
format_stamp: "Formatted at 2025-09-01 20:01:22 on dist-test-slave-2s4d"
server_key: "6a10c67860d517889c30e693ee2b2366"
server_key_iv: "753f490d26c73cbeb7b25b6b026a1be0"
server_key_version: "encryptionkey@0"
I20250901 20:01:22.612087 15957 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:01:22.613930 15972 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:22.614820 15957 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:22.614916 15957 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "0ac688c1d743428196b1ba6504dc7ca1"
format_stamp: "Formatted at 2025-09-01 20:01:22 on dist-test-slave-2s4d"
server_key: "6a10c67860d517889c30e693ee2b2366"
server_key_iv: "753f490d26c73cbeb7b25b6b026a1be0"
server_key_version: "encryptionkey@0"
I20250901 20:01:22.614992 15957 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:22.646881 15957 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:22.649498 15957 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:22.649667 15957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:22.651147 15957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:22.651201 15957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:22.651257 15957 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:22.651290 15957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 01 20:01:22 dist-test-slave-2s4d krb5kdc[15693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:22.662392 15957 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:34423
I20250901 20:01:22.662488 16087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:34423 every 8 connection(s)
I20250901 20:01:22.662881 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.651845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:45153 (local address 127.14.67.126:38487)
0901 20:01:22.651988 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:22.651993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:22.652746 (+ 753us) server_negotiation.cc:407] Connection header received
0901 20:01:22.653842 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:22.653847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:22.653907 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:22.654008 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:22.655424 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.656008 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.656683 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.656859 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.659160 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:22.659183 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:22.659186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:22.659211 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:22.660746 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.661368 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.661372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.661374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.661426 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.661720 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.661725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.661727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.661917 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:22.662027 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:22.662612 (+ 585us) server_negotiation.cc:299] Negotiation successful
0901 20:01:22.662733 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250901 20:01:22.663363 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.652077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38487 (local address 127.14.67.66:45153)
0901 20:01:22.652584 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:22.652631 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:22.653592 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:22.654215 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:22.654227 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:22.654725 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:22.655240 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.655252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.656134 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.656138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:22.656551 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.656559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.656763 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.657403 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:22.657431 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:22.659012 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:22.660918 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:22.660924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:22.660934 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:22.661243 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:22.661526 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:22.661529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:22.661530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:22.661619 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:22.662027 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:22.662033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:22.662283 (+ 250us) client_negotiation.cc:769] Sending connection context
0901 20:01:22.662491 (+ 208us) client_negotiation.cc:240] Negotiation successful
0901 20:01:22.662761 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":118,"threads_started":1}
I20250901 20:01:22.663478 15957 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:01:22.664866 16088 heartbeater.cc:344] Connected to a master server at 127.14.67.126:38487
I20250901 20:01:22.665012 16088 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:22.665444 16088 heartbeater.cc:507] Master 127.14.67.126:38487 requested a full tablet report, sending...
I20250901 20:01:22.666311 15741 ts_manager.cc:194] Registered new tserver with Master: 0ac688c1d743428196b1ba6504dc7ca1 (127.14.67.66:34423)
I20250901 20:01:22.666934 15741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:45153
I20250901 20:01:22.670575 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 15957
I20250901 20:01:22.670711 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:01:22.670961 14605 external_mini_cluster.cc:1468] Setting key 403aec524aff3da2b61accb9c401094c
I20250901 20:01:22.672784 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.666695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38487 (local address 127.14.67.66:46015)
0901 20:01:22.666924 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:22.666937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:22.667025 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:22.667288 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:22.667293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:22.667465 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:22.667662 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.667668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.668643 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.668646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:22.669015 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.669022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.669114 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.669602 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:22.669614 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:22.669868 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:22.671559 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:22.671563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:22.671568 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:22.671727 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:22.671947 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:22.671952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:22.671954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:22.671995 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:22.672322 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:22.672326 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:22.672397 (+ 71us) client_negotiation.cc:769] Sending connection context
0901 20:01:22.672494 (+ 97us) client_negotiation.cc:240] Negotiation successful
0901 20:01:22.672612 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
I20250901 20:01:22.672786 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.666618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:46015 (local address 127.14.67.126:38487)
0901 20:01:22.666740 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:22.666745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:22.667039 (+ 294us) server_negotiation.cc:407] Connection header received
0901 20:01:22.667130 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:22.667133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:22.667178 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:22.667258 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:22.667782 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.668447 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.669138 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.669271 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.669985 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:22.669998 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:22.670000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:22.670024 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:22.671452 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.671827 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.671831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.671832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.671869 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.672072 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.672075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.672077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.672219 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:22.672319 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:22.672550 (+ 231us) server_negotiation.cc:299] Negotiation successful
0901 20:01:22.672645 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:22.711409 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:38487
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:36363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:22.811118 16099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:22.811292 16099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:22.811358 16099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:22.814353 16099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:22.814401 16099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:22.814476 16099 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250901 20:01:22.814568 16099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:01:22.818444 16099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:36363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.67.126:38487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:22.819409 16099 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:22.820160 16099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:22.826089 16104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:22.826082 16107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:22.826082 16105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:22.826318 16099 server_base.cc:1047] running on GCE node
I20250901 20:01:22.826751 16099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:22.827220 16099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:22.828365 16099 hybrid_clock.cc:648] HybridClock initialized: now 1756756882828360 us; error 27 us; skew 500 ppm
Sep 01 20:01:22 dist-test-slave-2s4d krb5kdc[15693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:22.830909 16099 init.cc:377] Logged in from keytab as kudu/127.14.67.67@KRBTEST.COM (short username kudu)
I20250901 20:01:22.831790 16099 webserver.cc:480] Webserver started at http://127.14.67.67:38711/ using document root <none> and password file <none>
I20250901 20:01:22.832288 16099 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:22.832334 16099 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:22.832499 16099 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:22.834059 16099 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "30287a3e0d224aa79cff3926ec85df7c"
format_stamp: "Formatted at 2025-09-01 20:01:22 on dist-test-slave-2s4d"
server_key: "f505192f360d6d1d9f99fb239683ca35"
server_key_iv: "1b5eeb38a461bd2a996694f916d242f7"
server_key_version: "encryptionkey@0"
I20250901 20:01:22.834493 16099 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "30287a3e0d224aa79cff3926ec85df7c"
format_stamp: "Formatted at 2025-09-01 20:01:22 on dist-test-slave-2s4d"
server_key: "f505192f360d6d1d9f99fb239683ca35"
server_key_iv: "1b5eeb38a461bd2a996694f916d242f7"
server_key_version: "encryptionkey@0"
I20250901 20:01:22.837605 16099 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20250901 20:01:22.839563 16114 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:22.840513 16099 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:01:22.840628 16099 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "30287a3e0d224aa79cff3926ec85df7c"
format_stamp: "Formatted at 2025-09-01 20:01:22 on dist-test-slave-2s4d"
server_key: "f505192f360d6d1d9f99fb239683ca35"
server_key_iv: "1b5eeb38a461bd2a996694f916d242f7"
server_key_version: "encryptionkey@0"
I20250901 20:01:22.840719 16099 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:22.858901 16099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:22.861353 16099 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:22.861519 16099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:22.862847 16099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:22.862895 16099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:22.862933 16099 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:22.862965 16099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 01 20:01:22 dist-test-slave-2s4d krb5kdc[15693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:22.873368 16099 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:40925
I20250901 20:01:22.873430 16229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:40925 every 8 connection(s)
I20250901 20:01:22.874226 16099 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:01:22.874492 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.863429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:40113 (local address 127.14.67.126:38487)
0901 20:01:22.863579 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:22.863584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:22.864254 (+ 670us) server_negotiation.cc:407] Connection header received
0901 20:01:22.865063 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:22.865067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:22.865133 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:22.865234 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:22.866774 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.867601 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.868332 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.868527 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.871035 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:22.871056 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:22.871060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:22.871091 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:22.872678 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.873164 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.873167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.873169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.873217 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.873539 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.873543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.873545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.873684 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:22.873776 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:22.874215 (+ 439us) server_negotiation.cc:299] Negotiation successful
0901 20:01:22.874324 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20250901 20:01:22.875084 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.863684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38487 (local address 127.14.67.67:40113)
0901 20:01:22.864107 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:22.864148 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:22.864876 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:22.865390 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:22.865402 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:22.865899 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:22.866618 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.866636 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.867732 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.867735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:22.868219 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.868232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.868416 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.868976 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:22.869011 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:22.870897 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:22.872780 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:22.872785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:22.872795 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:22.873050 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:22.873294 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:22.873299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:22.873301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:22.873405 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:22.873787 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:22.873795 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:22.873998 (+ 203us) client_negotiation.cc:769] Sending connection context
0901 20:01:22.874165 (+ 167us) client_negotiation.cc:240] Negotiation successful
0901 20:01:22.874399 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":100,"threads_started":1}
I20250901 20:01:22.876096 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 16099
I20250901 20:01:22.876210 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:01:22.876441 14605 external_mini_cluster.cc:1468] Setting key df2f33051c274737b5b3d109bca9e01f
I20250901 20:01:22.876708 16230 heartbeater.cc:344] Connected to a master server at 127.14.67.126:38487
I20250901 20:01:22.876883 16230 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:22.877419 16230 heartbeater.cc:507] Master 127.14.67.126:38487 requested a full tablet report, sending...
I20250901 20:01:22.878494 15740 ts_manager.cc:194] Registered new tserver with Master: 30287a3e0d224aa79cff3926ec85df7c (127.14.67.67:40925)
I20250901 20:01:22.879333 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250901 20:01:22.879316 15740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:40113
I20250901 20:01:22.884716 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.878240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:39309 (local address 127.14.67.126:38487)
0901 20:01:22.878865 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:22.878869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:22.878882 (+ 13us) server_negotiation.cc:407] Connection header received
0901 20:01:22.878913 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:22.878915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:22.878954 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:22.879029 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:22.879669 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.880392 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.881158 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.881362 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.882070 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:22.882085 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:22.882087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:22.882107 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:22.883468 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.883831 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.883836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.883838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.883874 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.884074 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.884078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.884079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.884195 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:22.884279 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:22.884502 (+ 223us) server_negotiation.cc:299] Negotiation successful
0901 20:01:22.884592 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532}
I20250901 20:01:22.884737 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.878166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38487 (local address 127.14.67.67:39309)
0901 20:01:22.878428 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:22.878443 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:22.878548 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:22.879118 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:22.879121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:22.879285 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:22.879534 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.879542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.880527 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.880530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:22.881051 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.881058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.881163 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.881711 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:22.881723 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:22.881962 (+ 239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:22.883571 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:22.883574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:22.883576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:22.883745 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:22.883951 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:22.883956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:22.883958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:22.883999 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:22.884279 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:22.884283 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:22.884353 (+ 70us) client_negotiation.cc:769] Sending connection context
0901 20:01:22.884439 (+ 86us) client_negotiation.cc:240] Negotiation successful
0901 20:01:22.884558 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
Sep 01 20:01:22 dist-test-slave-2s4d krb5kdc[15693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756882, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 01 20:01:22 dist-test-slave-2s4d krb5kdc[15693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756882, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:22.901050 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.894535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.14.67.126:38487)
0901 20:01:22.894654 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:22.894656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:22.894667 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:01:22.894696 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:22.894698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:22.894734 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:22.894802 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:22.895539 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.895985 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.896585 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.896699 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.898654 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:22.898668 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:22.898670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:22.898691 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:22.899966 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.900287 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.900289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.900291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.900322 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:22.900475 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:22.900477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:22.900478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:22.900584 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:22.900677 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:22.900817 (+ 140us) server_negotiation.cc:299] Negotiation successful
0901 20:01:22.900890 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250901 20:01:22.903221 15740 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 20:01:22.905454 15740 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:01:22.917028 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.912661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:38653 (local address 127.0.0.1:33736)
0901 20:01:22.912953 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:22.912992 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:22.913108 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:22.913882 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:22.913888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:22.913908 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:22.914227 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.914235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.915645 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.915648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:22.916577 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.916586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.916685 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.916731 (+ 46us) client_negotiation.cc:769] Sending connection context
0901 20:01:22.916809 (+ 78us) client_negotiation.cc:240] Negotiation successful
0901 20:01:22.916887 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
I20250901 20:01:22.917889 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.913957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.14.67.66:34423)
0901 20:01:22.914287 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:22.914295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:22.914630 (+ 335us) server_negotiation.cc:407] Connection header received
0901 20:01:22.914719 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:22.914725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:22.914872 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:22.914992 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:22.915308 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.916205 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.917150 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.917598 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.917682 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:22.917730 (+ 48us) server_negotiation.cc:299] Negotiation successful
0901 20:01:22.917786 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
I20250901 20:01:22.917212 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.913296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:40925 (local address 127.0.0.1:48784)
0901 20:01:22.913908 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:22.913920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:22.914014 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:22.914436 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:22.914441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:22.914466 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:22.914736 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.914743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.916102 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.916106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:22.916921 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.916929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.916997 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.917009 (+ 12us) client_negotiation.cc:769] Sending connection context
0901 20:01:22.917048 (+ 39us) client_negotiation.cc:240] Negotiation successful
0901 20:01:22.917103 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":91,"threads_started":1}
I20250901 20:01:22.918211 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.913426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.14.67.67:40925)
0901 20:01:22.913777 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:22.913782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:22.914013 (+ 231us) server_negotiation.cc:407] Connection header received
0901 20:01:22.914127 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:22.914131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:22.914295 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:22.914441 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:22.914842 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.915979 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.917340 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.917852 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.917943 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:22.918004 (+ 61us) server_negotiation.cc:299] Negotiation successful
0901 20:01:22.918075 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":93,"threads_started":1}
I20250901 20:01:22.918211 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.912800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.14.67.65:38653)
0901 20:01:22.913100 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:22.913105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:22.913126 (+ 21us) server_negotiation.cc:407] Connection header received
0901 20:01:22.913355 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:22.913367 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:22.913567 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:22.913716 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:22.914359 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.915506 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.917330 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.917852 (+ 522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.917943 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:22.918004 (+ 61us) server_negotiation.cc:299] Negotiation successful
0901 20:01:22.918078 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":120,"threads_started":1}
I20250901 20:01:22.917382 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:22.913609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:34423 (local address 127.0.0.1:38034)
0901 20:01:22.914427 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:22.914438 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:22.914537 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:22.914988 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:22.914991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:22.915007 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:22.915198 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.915205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.916339 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:22.916342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:22.917014 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:22.917023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:22.917134 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:22.917153 (+ 19us) client_negotiation.cc:769] Sending connection context
0901 20:01:22.917203 (+ 50us) client_negotiation.cc:240] Negotiation successful
0901 20:01:22.917262 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":75,"threads_started":1}
I20250901 20:01:22.920801 16164 tablet_service.cc:1468] Processing CreateTablet for tablet 37aef7ffffb04f12a41a1cffc1274049 (DEFAULT_TABLE table=test-table [id=719ed1a145ad4a309a23034422d9d5fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:22.920801 15880 tablet_service.cc:1468] Processing CreateTablet for tablet 37aef7ffffb04f12a41a1cffc1274049 (DEFAULT_TABLE table=test-table [id=719ed1a145ad4a309a23034422d9d5fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:22.921095 16022 tablet_service.cc:1468] Processing CreateTablet for tablet 37aef7ffffb04f12a41a1cffc1274049 (DEFAULT_TABLE table=test-table [id=719ed1a145ad4a309a23034422d9d5fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:22.921738 15880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37aef7ffffb04f12a41a1cffc1274049. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:22.921738 16164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37aef7ffffb04f12a41a1cffc1274049. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:22.921854 16022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37aef7ffffb04f12a41a1cffc1274049. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:22.926769 16253 tablet_bootstrap.cc:492] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1: Bootstrap starting.
I20250901 20:01:22.928156 16254 tablet_bootstrap.cc:492] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547: Bootstrap starting.
I20250901 20:01:22.929073 16253 tablet_bootstrap.cc:654] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:22.929684 16254 tablet_bootstrap.cc:654] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:22.930078 16253 log.cc:826] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:22.930297 16254 log.cc:826] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:22.930675 16255 tablet_bootstrap.cc:492] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c: Bootstrap starting.
I20250901 20:01:22.932293 16255 tablet_bootstrap.cc:654] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:22.932322 16254 tablet_bootstrap.cc:492] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547: No bootstrap required, opened a new log
I20250901 20:01:22.932468 16254 ts_tablet_manager.cc:1397] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250901 20:01:22.933212 16255 log.cc:826] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:22.934576 16253 tablet_bootstrap.cc:492] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1: No bootstrap required, opened a new log
I20250901 20:01:22.934762 16253 ts_tablet_manager.cc:1397] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250901 20:01:22.934854 16255 tablet_bootstrap.cc:492] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c: No bootstrap required, opened a new log
I20250901 20:01:22.935012 16255 ts_tablet_manager.cc:1397] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250901 20:01:22.935065 16254 raft_consensus.cc:357] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } }
I20250901 20:01:22.935273 16254 raft_consensus.cc:383] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:22.935328 16254 raft_consensus.cc:738] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da766224ead4bf59b16b202f3ab0547, State: Initialized, Role: FOLLOWER
I20250901 20:01:22.935735 16254 consensus_queue.cc:260] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } }
I20250901 20:01:22.936318 15946 heartbeater.cc:499] Master 127.14.67.126:38487 was elected leader, sending a full tablet report...
I20250901 20:01:22.936535 16254 ts_tablet_manager.cc:1428] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250901 20:01:22.937597 16255 raft_consensus.cc:357] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } }
I20250901 20:01:22.937595 16253 raft_consensus.cc:357] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } }
I20250901 20:01:22.937795 16253 raft_consensus.cc:383] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:22.937795 16255 raft_consensus.cc:383] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:22.937853 16253 raft_consensus.cc:738] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac688c1d743428196b1ba6504dc7ca1, State: Initialized, Role: FOLLOWER
I20250901 20:01:22.937860 16255 raft_consensus.cc:738] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30287a3e0d224aa79cff3926ec85df7c, State: Initialized, Role: FOLLOWER
I20250901 20:01:22.938264 16253 consensus_queue.cc:260] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } }
I20250901 20:01:22.938264 16255 consensus_queue.cc:260] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } }
I20250901 20:01:22.939064 16255 ts_tablet_manager.cc:1428] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20250901 20:01:22.939064 16253 ts_tablet_manager.cc:1428] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20250901 20:01:22.939378 16230 heartbeater.cc:499] Master 127.14.67.126:38487 was elected leader, sending a full tablet report...
I20250901 20:01:22.939553 16088 heartbeater.cc:499] Master 127.14.67.126:38487 was elected leader, sending a full tablet report...
W20250901 20:01:22.943320 15947 tablet.cc:2378] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 20:01:23.018818 16261 raft_consensus.cc:491] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:01:23.019007 16261 raft_consensus.cc:513] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } }
I20250901 20:01:23.020052 16261 leader_election.cc:290] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da766224ead4bf59b16b202f3ab0547 (127.14.67.65:38653), 30287a3e0d224aa79cff3926ec85df7c (127.14.67.67:40925)
I20250901 20:01:23.022765 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.020294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:38653 (local address 127.14.67.66:36455)
0901 20:01:23.020445 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:23.020461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:23.020567 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:23.020875 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:23.020878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:23.020898 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:23.021108 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.021114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.021804 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.021807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:23.022354 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.022360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.022479 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.022526 (+ 47us) client_negotiation.cc:769] Sending connection context
0901 20:01:23.022581 (+ 55us) client_negotiation.cc:240] Negotiation successful
0901 20:01:23.022639 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20250901 20:01:23.023198 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.020379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:36455 (local address 127.14.67.65:38653)
0901 20:01:23.020536 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.020542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.020558 (+ 16us) server_negotiation.cc:407] Connection header received
0901 20:01:23.020690 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.020694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.020757 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.020852 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:23.021222 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.021684 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.022471 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.022988 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.023028 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.023069 (+ 41us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.023113 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20250901 20:01:23.023661 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.020556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:40925 (local address 127.14.67.66:48935)
0901 20:01:23.020853 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:23.020866 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:23.020962 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:23.021186 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:23.021189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:23.021204 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:23.021440 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.021446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.022141 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.022144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:23.022777 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.022785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.023379 (+ 594us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.023397 (+ 18us) client_negotiation.cc:769] Sending connection context
0901 20:01:23.023468 (+ 71us) client_negotiation.cc:240] Negotiation successful
0901 20:01:23.023524 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":107,"threads_started":1}
I20250901 20:01:23.023669 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.020701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:48935 (local address 127.14.67.67:40925)
0901 20:01:23.020828 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.020832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.020942 (+ 110us) server_negotiation.cc:407] Connection header received
0901 20:01:23.021047 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.021050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.021096 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.021161 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:23.021559 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.022028 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.022923 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.023280 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.023324 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.023508 (+ 184us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.023567 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20250901 20:01:23.023689 15900 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37aef7ffffb04f12a41a1cffc1274049" candidate_uuid: "0ac688c1d743428196b1ba6504dc7ca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da766224ead4bf59b16b202f3ab0547" is_pre_election: true
I20250901 20:01:23.023981 15900 raft_consensus.cc:2466] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac688c1d743428196b1ba6504dc7ca1 in term 0.
I20250901 20:01:23.024355 16184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37aef7ffffb04f12a41a1cffc1274049" candidate_uuid: "0ac688c1d743428196b1ba6504dc7ca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30287a3e0d224aa79cff3926ec85df7c" is_pre_election: true
I20250901 20:01:23.024511 15976 leader_election.cc:304] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac688c1d743428196b1ba6504dc7ca1, 3da766224ead4bf59b16b202f3ab0547; no voters:
I20250901 20:01:23.024583 16184 raft_consensus.cc:2466] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac688c1d743428196b1ba6504dc7ca1 in term 0.
I20250901 20:01:23.024785 16261 raft_consensus.cc:2802] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:01:23.024866 16261 raft_consensus.cc:491] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:01:23.024928 16261 raft_consensus.cc:3058] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:23.025818 16261 raft_consensus.cc:513] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } }
I20250901 20:01:23.026124 16261 leader_election.cc:290] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [CANDIDATE]: Term 1 election: Requested vote from peers 3da766224ead4bf59b16b202f3ab0547 (127.14.67.65:38653), 30287a3e0d224aa79cff3926ec85df7c (127.14.67.67:40925)
I20250901 20:01:23.026479 15900 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37aef7ffffb04f12a41a1cffc1274049" candidate_uuid: "0ac688c1d743428196b1ba6504dc7ca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da766224ead4bf59b16b202f3ab0547"
I20250901 20:01:23.026547 16184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37aef7ffffb04f12a41a1cffc1274049" candidate_uuid: "0ac688c1d743428196b1ba6504dc7ca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30287a3e0d224aa79cff3926ec85df7c"
I20250901 20:01:23.026620 15900 raft_consensus.cc:3058] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:23.026680 16184 raft_consensus.cc:3058] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:23.027513 15900 raft_consensus.cc:2466] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac688c1d743428196b1ba6504dc7ca1 in term 1.
I20250901 20:01:23.027674 16184 raft_consensus.cc:2466] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac688c1d743428196b1ba6504dc7ca1 in term 1.
I20250901 20:01:23.027819 15976 leader_election.cc:304] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac688c1d743428196b1ba6504dc7ca1, 3da766224ead4bf59b16b202f3ab0547; no voters:
I20250901 20:01:23.027984 16261 raft_consensus.cc:2802] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:23.028196 16261 raft_consensus.cc:695] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac688c1d743428196b1ba6504dc7ca1, State: Running, Role: LEADER
I20250901 20:01:23.028483 16261 consensus_queue.cc:237] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } }
I20250901 20:01:23.031371 15739 catalog_manager.cc:5582] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac688c1d743428196b1ba6504dc7ca1 (127.14.67.66). New cstate: current_term: 1 leader_uuid: "0ac688c1d743428196b1ba6504dc7ca1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } health_report { overall_health: HEALTHY } } }
I20250901 20:01:23.070042 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.067355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38044 (local address 127.14.67.66:34423)
0901 20:01:23.067498 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.067502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.067514 (+ 12us) server_negotiation.cc:407] Connection header received
0901 20:01:23.067642 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.067645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.067694 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.067769 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:23.068069 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.068555 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.069124 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.069251 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.069295 (+ 44us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:23.069701 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:23.069783 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.069860 (+ 77us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.069901 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250901 20:01:23.074936 16184 raft_consensus.cc:1273] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c [term 1 FOLLOWER]: Refusing update from remote peer 0ac688c1d743428196b1ba6504dc7ca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:23.074961 15900 raft_consensus.cc:1273] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547 [term 1 FOLLOWER]: Refusing update from remote peer 0ac688c1d743428196b1ba6504dc7ca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:23.075484 16261 consensus_queue.cc:1035] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:23.075634 16263 consensus_queue.cc:1035] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:23.082825 16268 mvcc.cc:204] Tried to move back new op lower bound from 7195676193068277760 to 7195676192887132160. Current Snapshot: MvccSnapshot[applied={T|T < 7195676193068277760}]
I20250901 20:01:23.093667 15740 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46218:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:23.093797 15740 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46218:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:23.096024 15740 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949: Sending DeleteTablet for 3 replicas of tablet 37aef7ffffb04f12a41a1cffc1274049
I20250901 20:01:23.096663 15880 tablet_service.cc:1515] Processing DeleteTablet for tablet 37aef7ffffb04f12a41a1cffc1274049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:23 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:33736
I20250901 20:01:23.096800 16164 tablet_service.cc:1515] Processing DeleteTablet for tablet 37aef7ffffb04f12a41a1cffc1274049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:23 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:48784
I20250901 20:01:23.096891 16022 tablet_service.cc:1515] Processing DeleteTablet for tablet 37aef7ffffb04f12a41a1cffc1274049 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:23 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:38034
I20250901 20:01:23.097136 16276 tablet_replica.cc:331] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547: stopping tablet replica
I20250901 20:01:23.097306 16278 tablet_replica.cc:331] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1: stopping tablet replica
I20250901 20:01:23.097384 16276 raft_consensus.cc:2241] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:23.097497 16278 raft_consensus.cc:2241] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:01:23.097651 16276 raft_consensus.cc:2270] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:23.097807 16278 raft_consensus.cc:2270] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:23.097975 15740 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 20:01:23.098352 15740 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:01:23.099035 16278 ts_tablet_manager.cc:1905] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:23.099380 16277 tablet_replica.cc:331] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c: stopping tablet replica
I20250901 20:01:23.099623 16277 raft_consensus.cc:2241] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:23.099918 16277 raft_consensus.cc:2270] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:23.100126 16276 ts_tablet_manager.cc:1905] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:23.102468 16278 ts_tablet_manager.cc:1918] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:01:23.102473 16276 ts_tablet_manager.cc:1918] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:01:23.102574 16276 log.cc:1199] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/wal/wals/37aef7ffffb04f12a41a1cffc1274049
I20250901 20:01:23.102602 16278 log.cc:1199] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/37aef7ffffb04f12a41a1cffc1274049
I20250901 20:01:23.102896 16276 ts_tablet_manager.cc:1939] T 37aef7ffffb04f12a41a1cffc1274049 P 3da766224ead4bf59b16b202f3ab0547: Deleting consensus metadata
I20250901 20:01:23.102967 16278 ts_tablet_manager.cc:1939] T 37aef7ffffb04f12a41a1cffc1274049 P 0ac688c1d743428196b1ba6504dc7ca1: Deleting consensus metadata
I20250901 20:01:23.103873 15728 catalog_manager.cc:4928] TS 3da766224ead4bf59b16b202f3ab0547 (127.14.67.65:38653): tablet 37aef7ffffb04f12a41a1cffc1274049 (table test-table [id=719ed1a145ad4a309a23034422d9d5fa]) successfully deleted
I20250901 20:01:23.104116 16164 tablet_service.cc:1468] Processing CreateTablet for tablet 8ffee04fcfae45328b6dbc5cd428da4b (DEFAULT_TABLE table=test-table [id=9c6ad88e270b42bb9b0b766c8a4bb289]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:23.104120 15880 tablet_service.cc:1468] Processing CreateTablet for tablet 8ffee04fcfae45328b6dbc5cd428da4b (DEFAULT_TABLE table=test-table [id=9c6ad88e270b42bb9b0b766c8a4bb289]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:23.104316 16164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ffee04fcfae45328b6dbc5cd428da4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:23.104352 15880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ffee04fcfae45328b6dbc5cd428da4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:23.105877 16255 tablet_bootstrap.cc:492] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c: Bootstrap starting.
I20250901 20:01:23.106369 16254 tablet_bootstrap.cc:492] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547: Bootstrap starting.
I20250901 20:01:23.106937 16255 tablet_bootstrap.cc:654] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:23.107167 16277 ts_tablet_manager.cc:1905] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:23.107321 16254 tablet_bootstrap.cc:654] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:23.107584 15728 catalog_manager.cc:4928] TS 0ac688c1d743428196b1ba6504dc7ca1 (127.14.67.66:34423): tablet 37aef7ffffb04f12a41a1cffc1274049 (table test-table [id=719ed1a145ad4a309a23034422d9d5fa]) successfully deleted
I20250901 20:01:23.108464 16255 tablet_bootstrap.cc:492] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c: No bootstrap required, opened a new log
I20250901 20:01:23.108548 16255 ts_tablet_manager.cc:1397] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250901 20:01:23.108691 16022 tablet_service.cc:1468] Processing CreateTablet for tablet 8ffee04fcfae45328b6dbc5cd428da4b (DEFAULT_TABLE table=test-table [id=9c6ad88e270b42bb9b0b766c8a4bb289]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:23.108937 16022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ffee04fcfae45328b6dbc5cd428da4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:23.109032 16255 raft_consensus.cc:357] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.109153 16255 raft_consensus.cc:383] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:23.109189 16255 raft_consensus.cc:738] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30287a3e0d224aa79cff3926ec85df7c, State: Initialized, Role: FOLLOWER
I20250901 20:01:23.109316 16255 consensus_queue.cc:260] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.109611 16255 ts_tablet_manager.cc:1428] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250901 20:01:23.110675 16277 ts_tablet_manager.cc:1918] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:01:23.110801 16277 log.cc:1199] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/37aef7ffffb04f12a41a1cffc1274049
I20250901 20:01:23.111116 16277 ts_tablet_manager.cc:1939] T 37aef7ffffb04f12a41a1cffc1274049 P 30287a3e0d224aa79cff3926ec85df7c: Deleting consensus metadata
I20250901 20:01:23.111405 16253 tablet_bootstrap.cc:492] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1: Bootstrap starting.
I20250901 20:01:23.111980 16254 tablet_bootstrap.cc:492] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547: No bootstrap required, opened a new log
I20250901 20:01:23.112064 16254 ts_tablet_manager.cc:1397] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250901 20:01:23.112401 16253 tablet_bootstrap.cc:654] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:23.112553 16254 raft_consensus.cc:357] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.112655 16254 raft_consensus.cc:383] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:23.112690 16254 raft_consensus.cc:738] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da766224ead4bf59b16b202f3ab0547, State: Initialized, Role: FOLLOWER
I20250901 20:01:23.112802 16254 consensus_queue.cc:260] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.113044 16254 ts_tablet_manager.cc:1428] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:01:23.113615 15728 catalog_manager.cc:4928] TS 30287a3e0d224aa79cff3926ec85df7c (127.14.67.67:40925): tablet 37aef7ffffb04f12a41a1cffc1274049 (table test-table [id=719ed1a145ad4a309a23034422d9d5fa]) successfully deleted
I20250901 20:01:23.114601 16253 tablet_bootstrap.cc:492] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1: No bootstrap required, opened a new log
I20250901 20:01:23.114681 16253 ts_tablet_manager.cc:1397] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250901 20:01:23.115099 16253 raft_consensus.cc:357] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.115198 16253 raft_consensus.cc:383] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:23.115229 16253 raft_consensus.cc:738] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac688c1d743428196b1ba6504dc7ca1, State: Initialized, Role: FOLLOWER
I20250901 20:01:23.115334 16253 consensus_queue.cc:260] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.115556 16253 ts_tablet_manager.cc:1428] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250901 20:01:23.125555 16231 tablet.cc:2378] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 20:01:23.148157 16260 raft_consensus.cc:491] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:01:23.148283 16260 raft_consensus.cc:513] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.149489 16260 leader_election.cc:290] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da766224ead4bf59b16b202f3ab0547 (127.14.67.65:38653), 0ac688c1d743428196b1ba6504dc7ca1 (127.14.67.66:34423)
I20250901 20:01:23.152704 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.149698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:38653 (local address 127.14.67.67:49141)
0901 20:01:23.149829 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:23.149846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:23.149945 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:23.150271 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:23.150274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:23.150294 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:23.150580 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.150588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.151610 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.151614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:23.152335 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.152343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.152453 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.152475 (+ 22us) client_negotiation.cc:769] Sending connection context
0901 20:01:23.152528 (+ 53us) client_negotiation.cc:240] Negotiation successful
0901 20:01:23.152583 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20250901 20:01:23.153180 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.149798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:49141 (local address 127.14.67.65:38653)
0901 20:01:23.149944 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.149949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.149965 (+ 16us) server_negotiation.cc:407] Connection header received
0901 20:01:23.150036 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.150039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.150090 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.150160 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:23.150723 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.151475 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.152452 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.152940 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.152984 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.153034 (+ 50us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.153097 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250901 20:01:23.153589 15900 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ffee04fcfae45328b6dbc5cd428da4b" candidate_uuid: "30287a3e0d224aa79cff3926ec85df7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da766224ead4bf59b16b202f3ab0547" is_pre_election: true
I20250901 20:01:23.153739 15900 raft_consensus.cc:2466] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30287a3e0d224aa79cff3926ec85df7c in term 0.
I20250901 20:01:23.153824 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.149953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:34423 (local address 127.14.67.67:38975)
0901 20:01:23.150256 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:23.150270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:23.150350 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:23.150760 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:23.150763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:23.150780 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:23.151048 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.151059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.151950 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.151953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:23.152868 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.152878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.153545 (+ 667us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.153561 (+ 16us) client_negotiation.cc:769] Sending connection context
0901 20:01:23.153635 (+ 74us) client_negotiation.cc:240] Negotiation successful
0901 20:01:23.153698 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":68,"threads_started":1}
I20250901 20:01:23.153831 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.150207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:38975 (local address 127.14.67.66:34423)
0901 20:01:23.150335 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.150339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.150352 (+ 13us) server_negotiation.cc:407] Connection header received
0901 20:01:23.150485 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.150491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.150570 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.150626 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:23.151209 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.151842 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.153008 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.153495 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.153527 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.153658 (+ 131us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.153719 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250901 20:01:23.154201 16118 leader_election.cc:304] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30287a3e0d224aa79cff3926ec85df7c, 3da766224ead4bf59b16b202f3ab0547; no voters:
I20250901 20:01:23.154387 16260 raft_consensus.cc:2802] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:01:23.154457 16260 raft_consensus.cc:491] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:01:23.154529 16260 raft_consensus.cc:3058] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:23.154795 16042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ffee04fcfae45328b6dbc5cd428da4b" candidate_uuid: "30287a3e0d224aa79cff3926ec85df7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac688c1d743428196b1ba6504dc7ca1" is_pre_election: true
I20250901 20:01:23.154999 16042 raft_consensus.cc:2466] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30287a3e0d224aa79cff3926ec85df7c in term 0.
I20250901 20:01:23.155689 16260 raft_consensus.cc:513] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.155999 16260 leader_election.cc:290] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [CANDIDATE]: Term 1 election: Requested vote from peers 3da766224ead4bf59b16b202f3ab0547 (127.14.67.65:38653), 0ac688c1d743428196b1ba6504dc7ca1 (127.14.67.66:34423)
I20250901 20:01:23.156297 15900 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ffee04fcfae45328b6dbc5cd428da4b" candidate_uuid: "30287a3e0d224aa79cff3926ec85df7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da766224ead4bf59b16b202f3ab0547"
I20250901 20:01:23.156320 16042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ffee04fcfae45328b6dbc5cd428da4b" candidate_uuid: "30287a3e0d224aa79cff3926ec85df7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac688c1d743428196b1ba6504dc7ca1"
I20250901 20:01:23.156407 15900 raft_consensus.cc:3058] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:23.156407 16042 raft_consensus.cc:3058] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:23.157493 15900 raft_consensus.cc:2466] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30287a3e0d224aa79cff3926ec85df7c in term 1.
I20250901 20:01:23.157807 16118 leader_election.cc:304] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30287a3e0d224aa79cff3926ec85df7c, 3da766224ead4bf59b16b202f3ab0547; no voters:
I20250901 20:01:23.158013 16042 raft_consensus.cc:2466] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30287a3e0d224aa79cff3926ec85df7c in term 1.
I20250901 20:01:23.158177 16260 raft_consensus.cc:2802] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:23.158375 16260 raft_consensus.cc:695] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 1 LEADER]: Becoming Leader. State: Replica: 30287a3e0d224aa79cff3926ec85df7c, State: Running, Role: LEADER
I20250901 20:01:23.158625 16260 consensus_queue.cc:237] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.160555 15740 catalog_manager.cc:5582] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 30287a3e0d224aa79cff3926ec85df7c (127.14.67.67). New cstate: current_term: 1 leader_uuid: "30287a3e0d224aa79cff3926ec85df7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } health_report { overall_health: HEALTHY } } }
W20250901 20:01:23.164409 16089 tablet.cc:2378] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 01 20:01:23 dist-test-slave-2s4d krb5kdc[15693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:23.199254 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.192053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38487 (local address 127.0.0.1:46226)
0901 20:01:23.192333 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:23.192345 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:23.192446 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:23.192683 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:23.192686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:23.193064 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:23.193241 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.193247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.193930 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.193933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:23.194323 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.194330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.194428 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.195082 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:23.195094 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:23.196394 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:23.197959 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:23.197966 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:23.197971 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:23.198158 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:23.198359 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:23.198362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:23.198364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:23.198402 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:23.198739 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:23.198747 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:23.198842 (+ 95us) client_negotiation.cc:769] Sending connection context
0901 20:01:23.198933 (+ 91us) client_negotiation.cc:240] Negotiation successful
0901 20:01:23.199065 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":81,"threads_started":1}
I20250901 20:01:23.199254 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.192104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.14.67.126:38487)
0901 20:01:23.192203 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.192206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.192429 (+ 223us) server_negotiation.cc:407] Connection header received
0901 20:01:23.192537 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.192540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.192578 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.192644 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:23.193363 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.193835 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.194431 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.194644 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.196509 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:23.196522 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:23.196524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:23.196543 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:23.197860 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:23.198243 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:23.198248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:23.198250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:23.198285 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:23.198475 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:23.198478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:23.198480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:23.198639 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:23.198729 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.198986 (+ 257us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.199114 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":34}
I20250901 20:01:23.203040 15740 catalog_manager.cc:2232] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:46226:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250901 20:01:23.203764 15740 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:01:23.208161 16022 tablet_service.cc:1468] Processing CreateTablet for tablet b437bd08e6ae4008b538cd68d719f20e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d4dee38d52b4c2faed4fab8464c4e7c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250901 20:01:23.208230 15880 tablet_service.cc:1468] Processing CreateTablet for tablet b437bd08e6ae4008b538cd68d719f20e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d4dee38d52b4c2faed4fab8464c4e7c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250901 20:01:23.208496 16022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b437bd08e6ae4008b538cd68d719f20e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:23.208570 15880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b437bd08e6ae4008b538cd68d719f20e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:23.210429 16253 tablet_bootstrap.cc:492] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1: Bootstrap starting.
I20250901 20:01:23.211194 16253 tablet_bootstrap.cc:654] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:23.211208 16254 tablet_bootstrap.cc:492] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: Bootstrap starting.
I20250901 20:01:23.211668 16164 tablet_service.cc:1468] Processing CreateTablet for tablet b437bd08e6ae4008b538cd68d719f20e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d4dee38d52b4c2faed4fab8464c4e7c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250901 20:01:23.212005 16164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b437bd08e6ae4008b538cd68d719f20e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:23.212121 16254 tablet_bootstrap.cc:654] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:23.213372 16254 tablet_bootstrap.cc:492] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: No bootstrap required, opened a new log
I20250901 20:01:23.213372 16253 tablet_bootstrap.cc:492] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1: No bootstrap required, opened a new log
I20250901 20:01:23.213460 16254 ts_tablet_manager.cc:1397] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:23.213451 16253 ts_tablet_manager.cc:1397] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250901 20:01:23.213919 16253 raft_consensus.cc:357] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.213919 16254 raft_consensus.cc:357] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.214016 16253 raft_consensus.cc:383] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:23.214016 16254 raft_consensus.cc:383] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:23.214051 16254 raft_consensus.cc:738] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da766224ead4bf59b16b202f3ab0547, State: Initialized, Role: FOLLOWER
I20250901 20:01:23.214051 16253 raft_consensus.cc:738] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac688c1d743428196b1ba6504dc7ca1, State: Initialized, Role: FOLLOWER
I20250901 20:01:23.214165 16253 consensus_queue.cc:260] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.214165 16254 consensus_queue.cc:260] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.214409 16253 ts_tablet_manager.cc:1428] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:01:23.214409 16254 ts_tablet_manager.cc:1428] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:01:23.214644 16263 tablet_replica.cc:440] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } }
I20250901 20:01:23.214680 16259 tablet_replica.cc:440] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } }
I20250901 20:01:23.214763 16263 tablet_replica.cc:443] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:23.214781 16259 tablet_replica.cc:443] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:23.215107 16255 tablet_bootstrap.cc:492] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c: Bootstrap starting.
I20250901 20:01:23.216578 16255 tablet_bootstrap.cc:654] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:23.217873 16255 tablet_bootstrap.cc:492] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c: No bootstrap required, opened a new log
I20250901 20:01:23.217957 16255 ts_tablet_manager.cc:1397] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250901 20:01:23.218402 16255 raft_consensus.cc:357] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.218580 16255 raft_consensus.cc:383] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:23.218626 16255 raft_consensus.cc:738] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30287a3e0d224aa79cff3926ec85df7c, State: Initialized, Role: FOLLOWER
I20250901 20:01:23.218741 16255 consensus_queue.cc:260] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.219063 16260 tablet_replica.cc:440] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } }
I20250901 20:01:23.219159 16260 tablet_replica.cc:443] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:23.219434 16255 ts_tablet_manager.cc:1428] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:01:23.466467 16259 raft_consensus.cc:491] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:01:23.466615 16259 raft_consensus.cc:513] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.467535 16259 leader_election.cc:290] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ac688c1d743428196b1ba6504dc7ca1 (127.14.67.66:34423), 30287a3e0d224aa79cff3926ec85df7c (127.14.67.67:40925)
I20250901 20:01:23.470613 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.467874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:34423 (local address 127.14.67.65:46609)
0901 20:01:23.468179 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:23.468193 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:23.468348 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:23.468670 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:23.468673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:23.468691 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:23.468921 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.468927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.469604 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.469607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:23.470173 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.470180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.470299 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.470321 (+ 22us) client_negotiation.cc:769] Sending connection context
0901 20:01:23.470375 (+ 54us) client_negotiation.cc:240] Negotiation successful
0901 20:01:23.470429 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
I20250901 20:01:23.470775 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.468221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:40925 (local address 127.14.67.65:57065)
0901 20:01:23.468510 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:23.468523 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:23.468627 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:23.468861 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:23.468864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:23.468881 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:23.469096 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.469102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.469766 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.469769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:23.470311 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.470318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.470421 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.470439 (+ 18us) client_negotiation.cc:769] Sending connection context
0901 20:01:23.470489 (+ 50us) client_negotiation.cc:240] Negotiation successful
0901 20:01:23.470569 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
I20250901 20:01:23.471061 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.467975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:46609 (local address 127.14.67.66:34423)
0901 20:01:23.468096 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.468099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.468261 (+ 162us) server_negotiation.cc:407] Connection header received
0901 20:01:23.468523 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.468526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.468572 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.468641 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:23.469043 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.469493 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.470299 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.470831 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.470876 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.470931 (+ 55us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.470980 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250901 20:01:23.471177 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.468319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:57065 (local address 127.14.67.67:40925)
0901 20:01:23.468438 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.468442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.468592 (+ 150us) server_negotiation.cc:407] Connection header received
0901 20:01:23.468714 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.468719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.468768 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.468835 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:23.469197 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.469651 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.470418 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.470915 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.470960 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.471014 (+ 54us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.471062 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250901 20:01:23.471437 16042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b437bd08e6ae4008b538cd68d719f20e" candidate_uuid: "3da766224ead4bf59b16b202f3ab0547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac688c1d743428196b1ba6504dc7ca1" is_pre_election: true
I20250901 20:01:23.471565 16042 raft_consensus.cc:2466] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da766224ead4bf59b16b202f3ab0547 in term 0.
I20250901 20:01:23.471577 16184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b437bd08e6ae4008b538cd68d719f20e" candidate_uuid: "3da766224ead4bf59b16b202f3ab0547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30287a3e0d224aa79cff3926ec85df7c" is_pre_election: true
I20250901 20:01:23.471726 16184 raft_consensus.cc:2466] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3da766224ead4bf59b16b202f3ab0547 in term 0.
I20250901 20:01:23.471909 15834 leader_election.cc:304] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac688c1d743428196b1ba6504dc7ca1, 3da766224ead4bf59b16b202f3ab0547; no voters:
I20250901 20:01:23.472090 16259 raft_consensus.cc:2802] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:01:23.472143 16259 raft_consensus.cc:491] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:01:23.472172 16259 raft_consensus.cc:3058] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:23.472993 16259 raft_consensus.cc:513] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.473284 16259 leader_election.cc:290] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [CANDIDATE]: Term 1 election: Requested vote from peers 0ac688c1d743428196b1ba6504dc7ca1 (127.14.67.66:34423), 30287a3e0d224aa79cff3926ec85df7c (127.14.67.67:40925)
I20250901 20:01:23.473628 16042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b437bd08e6ae4008b538cd68d719f20e" candidate_uuid: "3da766224ead4bf59b16b202f3ab0547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac688c1d743428196b1ba6504dc7ca1"
I20250901 20:01:23.473655 16184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b437bd08e6ae4008b538cd68d719f20e" candidate_uuid: "3da766224ead4bf59b16b202f3ab0547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30287a3e0d224aa79cff3926ec85df7c"
I20250901 20:01:23.473729 16042 raft_consensus.cc:3058] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:23.473759 16184 raft_consensus.cc:3058] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:23.474529 16042 raft_consensus.cc:2466] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da766224ead4bf59b16b202f3ab0547 in term 1.
I20250901 20:01:23.474570 16184 raft_consensus.cc:2466] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3da766224ead4bf59b16b202f3ab0547 in term 1.
I20250901 20:01:23.474822 15834 leader_election.cc:304] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac688c1d743428196b1ba6504dc7ca1, 3da766224ead4bf59b16b202f3ab0547; no voters:
I20250901 20:01:23.474983 16259 raft_consensus.cc:2802] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:23.475147 16259 raft_consensus.cc:695] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [term 1 LEADER]: Becoming Leader. State: Replica: 3da766224ead4bf59b16b202f3ab0547, State: Running, Role: LEADER
I20250901 20:01:23.475358 16259 consensus_queue.cc:237] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } }
I20250901 20:01:23.476374 16296 tablet_replica.cc:440] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: TxnStatusTablet state changed. Reason: New leader 3da766224ead4bf59b16b202f3ab0547. Latest consensus state: current_term: 1 leader_uuid: "3da766224ead4bf59b16b202f3ab0547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } }
I20250901 20:01:23.476476 16296 tablet_replica.cc:443] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: This TxnStatusTablet replica's current role is: LEADER
I20250901 20:01:23.476785 16298 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250901 20:01:23.476922 16298 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250901 20:01:23.477506 15740 catalog_manager.cc:5582] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3da766224ead4bf59b16b202f3ab0547 (127.14.67.65). New cstate: current_term: 1 leader_uuid: "3da766224ead4bf59b16b202f3ab0547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20250901 20:01:23.516376 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.513405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:38653 (local address 127.0.0.1:33738)
0901 20:01:23.513553 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:23.513568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:23.513677 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:23.513975 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:23.513979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:23.513987 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0901 20:01:23.514169 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.514174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.514866 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.514870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:23.515269 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.515276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.515374 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.515414 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0901 20:01:23.516104 (+ 690us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0901 20:01:23.516120 (+ 16us) client_negotiation.cc:769] Sending connection context
0901 20:01:23.516186 (+ 66us) client_negotiation.cc:240] Negotiation successful
0901 20:01:23.516257 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20250901 20:01:23.516376 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.513546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.14.67.65:38653)
0901 20:01:23.513680 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.513684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.513699 (+ 15us) server_negotiation.cc:407] Connection header received
0901 20:01:23.513822 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.513824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.513877 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.513958 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:23.514270 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.514763 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.515371 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.515564 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.515613 (+ 49us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:23.516001 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:23.516105 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.516202 (+ 97us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.516247 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250901 20:01:23.518801 16184 raft_consensus.cc:1273] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c [term 1 FOLLOWER]: Refusing update from remote peer 3da766224ead4bf59b16b202f3ab0547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:23.518983 16042 raft_consensus.cc:1273] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1 [term 1 FOLLOWER]: Refusing update from remote peer 3da766224ead4bf59b16b202f3ab0547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:23.519389 16296 consensus_queue.cc:1035] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:23.519572 16259 consensus_queue.cc:1035] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:23.521031 16260 tablet_replica.cc:440] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c: TxnStatusTablet state changed. Reason: New leader 3da766224ead4bf59b16b202f3ab0547. Latest consensus state: current_term: 1 leader_uuid: "3da766224ead4bf59b16b202f3ab0547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } }
I20250901 20:01:23.521127 16260 tablet_replica.cc:443] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:23.521224 16263 tablet_replica.cc:440] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1: TxnStatusTablet state changed. Reason: New leader 3da766224ead4bf59b16b202f3ab0547. Latest consensus state: current_term: 1 leader_uuid: "3da766224ead4bf59b16b202f3ab0547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } }
I20250901 20:01:23.521306 16263 tablet_replica.cc:443] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:23.523075 16267 mvcc.cc:204] Tried to move back new op lower bound from 7195676194888704000 to 7195676194717081600. Current Snapshot: MvccSnapshot[applied={T|T < 7195676194888704000}]
I20250901 20:01:23.523172 16259 tablet_replica.cc:440] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3da766224ead4bf59b16b202f3ab0547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } }
I20250901 20:01:23.523244 16259 tablet_replica.cc:443] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: This TxnStatusTablet replica's current role is: LEADER
I20250901 20:01:23.523627 16296 tablet_replica.cc:440] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3da766224ead4bf59b16b202f3ab0547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } }
I20250901 20:01:23.523725 16296 tablet_replica.cc:443] T b437bd08e6ae4008b538cd68d719f20e P 3da766224ead4bf59b16b202f3ab0547: This TxnStatusTablet replica's current role is: LEADER
I20250901 20:01:23.524402 16260 tablet_replica.cc:440] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3da766224ead4bf59b16b202f3ab0547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } }
I20250901 20:01:23.524502 16260 tablet_replica.cc:443] T b437bd08e6ae4008b538cd68d719f20e P 30287a3e0d224aa79cff3926ec85df7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:23.525772 16263 tablet_replica.cc:440] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3da766224ead4bf59b16b202f3ab0547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 } } peers { permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 } } peers { permanent_uuid: "30287a3e0d224aa79cff3926ec85df7c" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40925 } } }
I20250901 20:01:23.525857 16263 tablet_replica.cc:443] T b437bd08e6ae4008b538cd68d719f20e P 0ac688c1d743428196b1ba6504dc7ca1: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:23.526080 16266 mvcc.cc:204] Tried to move back new op lower bound from 7195676194888704000 to 7195676194717081600. Current Snapshot: MvccSnapshot[applied={T|T < 7195676194888704000 or (T in {7195676194888704000})}]
I20250901 20:01:23.531461 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.528565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.14.67.67:40925)
0901 20:01:23.528666 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.528670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.528681 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:01:23.528818 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.528820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.528864 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.528930 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:23.529312 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.529752 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.530598 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.530723 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.530767 (+ 44us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:23.531144 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:23.531226 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.531307 (+ 81us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.531345 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250901 20:01:23.539697 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.537146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:33049 (local address 127.14.67.65:38653)
0901 20:01:23.537277 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.537280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.537363 (+ 83us) server_negotiation.cc:407] Connection header received
0901 20:01:23.537478 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.537481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.537518 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.537577 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:23.537934 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.538348 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.539186 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.539334 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.539368 (+ 34us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:23.539422 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:23.539481 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.539580 (+ 99us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.539613 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20250901 20:01:23.539769 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.537023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:38653 (local address 127.14.67.67:33049)
0901 20:01:23.537287 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:23.537300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:23.537396 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:23.537597 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:23.537600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:23.537608 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0901 20:01:23.537848 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.537855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.538430 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.538433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:23.539086 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.539094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.539194 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.539228 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0901 20:01:23.539503 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0901 20:01:23.539508 (+ 5us) client_negotiation.cc:769] Sending connection context
0901 20:01:23.539572 (+ 64us) client_negotiation.cc:240] Negotiation successful
0901 20:01:23.539631 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
I20250901 20:01:23.545934 16042 raft_consensus.cc:1273] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1 [term 1 FOLLOWER]: Refusing update from remote peer 30287a3e0d224aa79cff3926ec85df7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:23.545949 15900 raft_consensus.cc:1273] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547 [term 1 FOLLOWER]: Refusing update from remote peer 30287a3e0d224aa79cff3926ec85df7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:23.546346 16260 consensus_queue.cc:1035] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac688c1d743428196b1ba6504dc7ca1" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 34423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:23.546483 16283 consensus_queue.cc:1035] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da766224ead4bf59b16b202f3ab0547" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 38653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:23.566798 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.563527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:33321 (local address 127.14.67.67:40925)
0901 20:01:23.563645 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.563649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.563847 (+ 198us) server_negotiation.cc:407] Connection header received
0901 20:01:23.563967 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.563970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.564008 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.564067 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:23.564519 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.565031 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.565965 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.566152 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.566363 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:23.566425 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:23.566487 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.566657 (+ 170us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.566703 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250901 20:01:23.566823 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.563472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:40925 (local address 127.14.67.65:33321)
0901 20:01:23.563767 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:23.563782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:23.563884 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:23.564093 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:23.564096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:23.564114 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0901 20:01:23.564423 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.564429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.565120 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.565124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:23.565831 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:23.565841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.566218 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.566255 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0901 20:01:23.566539 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0901 20:01:23.566546 (+ 7us) client_negotiation.cc:769] Sending connection context
0901 20:01:23.566614 (+ 68us) client_negotiation.cc:240] Negotiation successful
0901 20:01:23.566677 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
I20250901 20:01:23.589672 15741 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46218:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:23.589825 15741 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46218:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:23.591877 15741 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 911d55fff6d2451e8356122292784949: Sending DeleteTablet for 3 replicas of tablet 8ffee04fcfae45328b6dbc5cd428da4b
I20250901 20:01:23.592685 16022 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ffee04fcfae45328b6dbc5cd428da4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:23 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:38034
I20250901 20:01:23.592840 16278 tablet_replica.cc:331] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1: stopping tablet replica
I20250901 20:01:23.592931 16278 raft_consensus.cc:2241] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:23.593019 16278 raft_consensus.cc:2270] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:23.593348 16164 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ffee04fcfae45328b6dbc5cd428da4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:23 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:48784
I20250901 20:01:23.593511 16277 tablet_replica.cc:331] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c: stopping tablet replica
I20250901 20:01:23.593617 16277 raft_consensus.cc:2241] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:01:23.593864 16277 raft_consensus.cc:2270] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:23.594786 16278 ts_tablet_manager.cc:1905] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:23.597259 16278 ts_tablet_manager.cc:1918] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:01:23.597330 16278 log.cc:1199] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/8ffee04fcfae45328b6dbc5cd428da4b
I20250901 20:01:23.597576 16278 ts_tablet_manager.cc:1939] T 8ffee04fcfae45328b6dbc5cd428da4b P 0ac688c1d743428196b1ba6504dc7ca1: Deleting consensus metadata
I20250901 20:01:23.597887 16277 ts_tablet_manager.cc:1905] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:23.598140 15728 catalog_manager.cc:4928] TS 0ac688c1d743428196b1ba6504dc7ca1 (127.14.67.66:34423): tablet 8ffee04fcfae45328b6dbc5cd428da4b (table test-table [id=9c6ad88e270b42bb9b0b766c8a4bb289]) successfully deleted
I20250901 20:01:23.598678 15880 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ffee04fcfae45328b6dbc5cd428da4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:23 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:33736
I20250901 20:01:23.600329 16277 ts_tablet_manager.cc:1918] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:01:23.600633 16276 tablet_replica.cc:331] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547: stopping tablet replica
I20250901 20:01:23.600764 16276 raft_consensus.cc:2241] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:23.600867 16276 raft_consensus.cc:2270] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:23.602017 16276 ts_tablet_manager.cc:1905] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:23.604485 16276 ts_tablet_manager.cc:1918] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:01:23.604563 16276 log.cc:1199] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-0/wal/wals/8ffee04fcfae45328b6dbc5cd428da4b
Sep 01 20:01:23 dist-test-slave-2s4d krb5kdc[15693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756882, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.67.65@KRBTEST.COM
I20250901 20:01:23.604830 16276 ts_tablet_manager.cc:1939] T 8ffee04fcfae45328b6dbc5cd428da4b P 3da766224ead4bf59b16b202f3ab0547: Deleting consensus metadata
I20250901 20:01:23.605544 15728 catalog_manager.cc:4928] TS 3da766224ead4bf59b16b202f3ab0547 (127.14.67.65:38653): tablet 8ffee04fcfae45328b6dbc5cd428da4b (table test-table [id=9c6ad88e270b42bb9b0b766c8a4bb289]) successfully deleted
I20250901 20:01:23.605746 16277 log.cc:1199] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/8ffee04fcfae45328b6dbc5cd428da4b
I20250901 20:01:23.606048 16277 ts_tablet_manager.cc:1939] T 8ffee04fcfae45328b6dbc5cd428da4b P 30287a3e0d224aa79cff3926ec85df7c: Deleting consensus metadata
I20250901 20:01:23.606832 15728 catalog_manager.cc:4928] TS 30287a3e0d224aa79cff3926ec85df7c (127.14.67.67:40925): tablet 8ffee04fcfae45328b6dbc5cd428da4b (table test-table [id=9c6ad88e270b42bb9b0b766c8a4bb289]) successfully deleted
I20250901 20:01:23.609133 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.594234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.14.67.65:38653)
0901 20:01:23.594341 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.594346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.599038 (+ 4692us) server_negotiation.cc:407] Connection header received
0901 20:01:23.599213 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.599217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.599260 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.599306 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:23.600433 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.601315 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.602214 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.602396 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.605392 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:23.605412 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:23.605423 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:23.605459 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:23.607427 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:23.607880 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:23.607885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:23.607888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:23.607945 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:23.608196 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:23.608200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:23.608202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:23.608386 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:23.608535 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.608874 (+ 339us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.608992 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20250901 20:01:23.609949 15940 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33750
I20250901 20:01:23.622586 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:23.611563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.14.67.126:38487)
0901 20:01:23.611679 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:23.611683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:23.614798 (+ 3115us) server_negotiation.cc:407] Connection header received
0901 20:01:23.614989 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:23.614993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:23.615052 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:23.615145 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:23.616049 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.616864 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:23.617730 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:23.617924 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:23.619146 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:23.619167 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:23.619171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:23.619199 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:23.620988 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:23.621412 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:23.621416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:23.621418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:23.621467 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:23.621697 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:23.621700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:23.621702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:23.621874 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:23.622014 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:23.622204 (+ 190us) server_negotiation.cc:299] Negotiation successful
0901 20:01:23.622333 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20250901 20:01:23.622939 15741 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46242
I20250901 20:01:23.624817 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 15815
I20250901 20:01:23.632072 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 15957
I20250901 20:01:23.639001 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 16099
I20250901 20:01:23.646723 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 15709
2025-09-01T20:01:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3770 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:01:23 dist-test-slave-2s4d krb5kdc[16320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:01:23 dist-test-slave-2s4d krb5kdc[16320](info): set up 2 sockets
Sep 01 20:01:23 dist-test-slave-2s4d krb5kdc[16320](info): commencing operation
krb5kdc: starting...
W20250901 20:01:25.619127 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.950s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:01:25 dist-test-slave-2s4d krb5kdc[16320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:01:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:01:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:25.743011 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:38019
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:38019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:25.843437 16336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:25.843645 16336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:25.843700 16336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:25.846781 16336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:01:25.846839 16336 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250901 20:01:25.846873 16336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:25.846915 16336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:25.846961 16336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:01:25.846994 16336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:01:25.851344 16336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:38019
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:38019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:25.852350 16336 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:25.853140 16336 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:25.858390 16344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:25.858472 16342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:25.858490 16341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:25.858758 16336 server_base.cc:1047] running on GCE node
I20250901 20:01:25.859192 16336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:25.859992 16336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:25.861137 16336 hybrid_clock.cc:648] HybridClock initialized: now 1756756885861116 us; error 38 us; skew 500 ppm
Sep 01 20:01:25 dist-test-slave-2s4d krb5kdc[16320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756885, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:25.863530 16336 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:01:25.864436 16336 webserver.cc:480] Webserver started at http://127.14.67.126:41515/ using document root <none> and password file <none>
I20250901 20:01:25.864910 16336 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:25.864954 16336 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:25.865106 16336 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:25.866740 16336 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "1b4080a8180f45afa29d779ceaacc1de"
format_stamp: "Formatted at 2025-09-01 20:01:25 on dist-test-slave-2s4d"
server_key: "5ccf6e2dd720e935d594b6d488ae824a"
server_key_iv: "00c182734d4c36b71cbf160280a1cbb3"
server_key_version: "encryptionkey@0"
I20250901 20:01:25.867172 16336 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "1b4080a8180f45afa29d779ceaacc1de"
format_stamp: "Formatted at 2025-09-01 20:01:25 on dist-test-slave-2s4d"
server_key: "5ccf6e2dd720e935d594b6d488ae824a"
server_key_iv: "00c182734d4c36b71cbf160280a1cbb3"
server_key_version: "encryptionkey@0"
I20250901 20:01:25.870134 16336 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250901 20:01:25.872063 16351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:25.872958 16336 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250901 20:01:25.873052 16336 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "1b4080a8180f45afa29d779ceaacc1de"
format_stamp: "Formatted at 2025-09-01 20:01:25 on dist-test-slave-2s4d"
server_key: "5ccf6e2dd720e935d594b6d488ae824a"
server_key_iv: "00c182734d4c36b71cbf160280a1cbb3"
server_key_version: "encryptionkey@0"
I20250901 20:01:25.873139 16336 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:25.889842 16336 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:25.892769 16336 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:25.892956 16336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:25.900261 16413 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:38019 every 8 connection(s)
I20250901 20:01:25.900265 16336 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:38019
I20250901 20:01:25.901232 16336 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:01:25.904026 16414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:25.907824 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 16336
I20250901 20:01:25.907933 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:01:25.908146 14605 external_mini_cluster.cc:1468] Setting key 76e54407fd0ac31fffbe9cfea284a860
I20250901 20:01:25.908850 16414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de: Bootstrap starting.
I20250901 20:01:25.911252 16414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:25.911989 16414 log.cc:826] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:25.913765 16414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de: No bootstrap required, opened a new log
Sep 01 20:01:25 dist-test-slave-2s4d krb5kdc[16320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:25.916278 16414 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4080a8180f45afa29d779ceaacc1de" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38019 } }
I20250901 20:01:25.916448 16414 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:25.916483 16414 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4080a8180f45afa29d779ceaacc1de, State: Initialized, Role: FOLLOWER
I20250901 20:01:25.916846 16414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4080a8180f45afa29d779ceaacc1de" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38019 } }
I20250901 20:01:25.916970 16414 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:01:25.917029 16414 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:01:25.917145 16414 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:25.917901 16414 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4080a8180f45afa29d779ceaacc1de" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38019 } }
I20250901 20:01:25.918192 16414 leader_election.cc:304] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b4080a8180f45afa29d779ceaacc1de; no voters:
I20250901 20:01:25.918469 16414 leader_election.cc:290] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:01:25.918571 16419 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:25.918778 16419 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [term 1 LEADER]: Becoming Leader. State: Replica: 1b4080a8180f45afa29d779ceaacc1de, State: Running, Role: LEADER
I20250901 20:01:25.919072 16419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4080a8180f45afa29d779ceaacc1de" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38019 } }
I20250901 20:01:25.919574 16414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:01:25.920120 16419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b4080a8180f45afa29d779ceaacc1de. Latest consensus state: current_term: 1 leader_uuid: "1b4080a8180f45afa29d779ceaacc1de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4080a8180f45afa29d779ceaacc1de" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38019 } } }
I20250901 20:01:25.920258 16419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:25.920460 16420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b4080a8180f45afa29d779ceaacc1de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4080a8180f45afa29d779ceaacc1de" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38019 } } }
I20250901 20:01:25.920553 16420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:25.921550 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:25.909273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.14.67.126:38019)
0901 20:01:25.909713 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:25.909724 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:25.909768 (+ 44us) server_negotiation.cc:407] Connection header received
0901 20:01:25.910413 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:25.910436 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:25.910817 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:25.911131 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:25.911920 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:25.912874 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:25.913520 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:25.913781 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:25.915935 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:25.915952 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:25.915962 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:25.915993 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:25.918368 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:25.918865 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:25.918870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:25.918875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:25.918958 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:25.919210 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:25.919213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:25.919214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:25.919589 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:25.919786 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:25.920511 (+ 725us) server_negotiation.cc:299] Negotiation successful
0901 20:01:25.920785 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":123,"threads_started":1}
W20250901 20:01:25.924566 16434 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250901 20:01:25.924638 16434 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250901 20:01:25.924738 16424 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:01:25.925433 16424 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:01:25.930205 16424 catalog_manager.cc:1349] Generated new cluster ID: ddd1e6841ca94ca3b3dce333a6e7c008
I20250901 20:01:25.930315 16424 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:01:25.957947 16424 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:01:25.958793 16424 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:01:25.967072 16424 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de: Generated new TSK 0
I20250901 20:01:25.967630 16424 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:26.015686 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:38019
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:26.139997 16442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:26.140162 16442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:26.140237 16442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:26.143226 16442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:26.143289 16442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:26.143340 16442 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250901 20:01:26.143361 16442 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250901 20:01:26.143404 16442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:01:26.147400 16442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.67.126:38019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:26.148227 16442 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:26.148902 16442 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:26.154735 16450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:26.154728 16447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:26.154731 16448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:26.154875 16442 server_base.cc:1047] running on GCE node
I20250901 20:01:26.155215 16442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:26.155737 16442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:26.156872 16442 hybrid_clock.cc:648] HybridClock initialized: now 1756756886156859 us; error 26 us; skew 500 ppm
Sep 01 20:01:26 dist-test-slave-2s4d krb5kdc[16320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756886, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:26.159319 16442 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:01:26.160207 16442 webserver.cc:480] Webserver started at http://127.14.67.65:37207/ using document root <none> and password file <none>
I20250901 20:01:26.160658 16442 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:26.160699 16442 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:26.160833 16442 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:26.162286 16442 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "6ca3083eb2e1425a828f5f42e283d36d"
format_stamp: "Formatted at 2025-09-01 20:01:26 on dist-test-slave-2s4d"
server_key: "dadbfb3698d3b8bae44471069c17e80f"
server_key_iv: "840ea41752db620503f6d55d22c9bb0f"
server_key_version: "encryptionkey@0"
I20250901 20:01:26.162742 16442 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "6ca3083eb2e1425a828f5f42e283d36d"
format_stamp: "Formatted at 2025-09-01 20:01:26 on dist-test-slave-2s4d"
server_key: "dadbfb3698d3b8bae44471069c17e80f"
server_key_iv: "840ea41752db620503f6d55d22c9bb0f"
server_key_version: "encryptionkey@0"
I20250901 20:01:26.165566 16442 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20250901 20:01:26.167479 16457 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:26.168337 16442 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:01:26.168433 16442 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "6ca3083eb2e1425a828f5f42e283d36d"
format_stamp: "Formatted at 2025-09-01 20:01:26 on dist-test-slave-2s4d"
server_key: "dadbfb3698d3b8bae44471069c17e80f"
server_key_iv: "840ea41752db620503f6d55d22c9bb0f"
server_key_version: "encryptionkey@0"
I20250901 20:01:26.168533 16442 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:26.189910 16442 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:26.192626 16442 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:26.192799 16442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:26.194062 16442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:26.194113 16442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:26.194166 16442 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:26.194200 16442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 01 20:01:26 dist-test-slave-2s4d krb5kdc[16320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756886, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:26.205130 16442 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:33477
I20250901 20:01:26.205200 16572 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:33477 every 8 connection(s)
I20250901 20:01:26.205850 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.194658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:59143 (local address 127.14.67.126:38019)
0901 20:01:26.194813 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.194818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.195544 (+ 726us) server_negotiation.cc:407] Connection header received
0901 20:01:26.196586 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.196590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.196644 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.196714 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:26.198373 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.199035 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.199652 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.199821 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.202215 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:26.202230 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:26.202233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:26.202256 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:26.204011 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.204511 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.204513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.204514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.204552 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.204839 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.204842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.204844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.205029 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:26.205127 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.205635 (+ 508us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.205735 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20250901 20:01:26.206153 16442 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:01:26.206569 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.194905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38019 (local address 127.14.67.65:59143)
0901 20:01:26.195360 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.195417 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.196363 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.196893 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.196902 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.197469 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:26.198214 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.198229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.199149 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.199152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.199557 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.199565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.199739 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.200237 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:26.200259 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:26.202079 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:26.204118 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.204123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.204133 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.204420 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.204637 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.204640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.204642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.204745 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.205130 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:26.205136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:26.205402 (+ 266us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.205602 (+ 200us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.205837 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":92,"threads_started":1}
I20250901 20:01:26.207919 16573 heartbeater.cc:344] Connected to a master server at 127.14.67.126:38019
I20250901 20:01:26.208060 16573 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:26.208482 16573 heartbeater.cc:507] Master 127.14.67.126:38019 requested a full tablet report, sending...
I20250901 20:01:26.209738 16368 ts_manager.cc:194] Registered new tserver with Master: 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477)
I20250901 20:01:26.210721 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 16442
I20250901 20:01:26.210836 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:01:26.211050 14605 external_mini_cluster.cc:1468] Setting key f0f1d11cb2f99290ce6e5b2cb63dc225
I20250901 20:01:26.211186 16368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:59143
I20250901 20:01:26.215603 16579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.209638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38019 (local address 127.14.67.65:51859)
0901 20:01:26.209949 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.209960 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.210036 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.210231 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.210234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.210361 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:26.210604 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.210611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.211248 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.211251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.211735 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.211744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.211848 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.212369 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:26.212384 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:26.212727 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:26.214383 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.214393 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.214395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.214599 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.214824 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.214827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.214829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.214881 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.215173 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:26.215179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:26.215260 (+ 81us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.215377 (+ 117us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.215488 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":80,"threads_started":1}
I20250901 20:01:26.215757 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.209702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:51859 (local address 127.14.67.126:38019)
0901 20:01:26.209821 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.209825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.210025 (+ 200us) server_negotiation.cc:407] Connection header received
0901 20:01:26.210112 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.210115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.210154 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.210225 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:26.210712 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.211148 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.211845 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.212032 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.212861 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:26.212876 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:26.212879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:26.212902 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:26.214283 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.214692 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.214696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.214700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.214737 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.214957 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.214961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.214963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.215077 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:26.215163 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.215485 (+ 322us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.215602 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:26.251344 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:38019
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:26.348613 16584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:26.348783 16584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:26.348838 16584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:26.351815 16584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:26.351862 16584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:26.351912 16584 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250901 20:01:26.351933 16584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250901 20:01:26.351970 16584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:01:26.356065 16584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.67.126:38019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:26.356904 16584 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:26.357620 16584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:26.363710 16589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:26.363690 16590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:26.363677 16592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:26.364244 16584 server_base.cc:1047] running on GCE node
I20250901 20:01:26.364534 16584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:26.364998 16584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:26.366143 16584 hybrid_clock.cc:648] HybridClock initialized: now 1756756886366140 us; error 37 us; skew 500 ppm
Sep 01 20:01:26 dist-test-slave-2s4d krb5kdc[16320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756886, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:26.368672 16584 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:01:26.369593 16584 webserver.cc:480] Webserver started at http://127.14.67.66:36741/ using document root <none> and password file <none>
I20250901 20:01:26.370090 16584 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:26.370141 16584 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:26.370311 16584 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:26.371992 16584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb"
format_stamp: "Formatted at 2025-09-01 20:01:26 on dist-test-slave-2s4d"
server_key: "187e388d05ccd83ecb77574bbab2bd00"
server_key_iv: "4deb6e71239fc3ec83515b64d7d91683"
server_key_version: "encryptionkey@0"
I20250901 20:01:26.372438 16584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb"
format_stamp: "Formatted at 2025-09-01 20:01:26 on dist-test-slave-2s4d"
server_key: "187e388d05ccd83ecb77574bbab2bd00"
server_key_iv: "4deb6e71239fc3ec83515b64d7d91683"
server_key_version: "encryptionkey@0"
I20250901 20:01:26.375517 16584 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250901 20:01:26.377573 16599 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:26.378480 16584 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:26.378612 16584 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb"
format_stamp: "Formatted at 2025-09-01 20:01:26 on dist-test-slave-2s4d"
server_key: "187e388d05ccd83ecb77574bbab2bd00"
server_key_iv: "4deb6e71239fc3ec83515b64d7d91683"
server_key_version: "encryptionkey@0"
I20250901 20:01:26.378708 16584 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:26.399344 16584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:26.401716 16584 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:26.401863 16584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:26.403180 16584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:26.403239 16584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:26.403290 16584 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:26.403319 16584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 01 20:01:26 dist-test-slave-2s4d krb5kdc[16320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756886, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:26.414237 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.403716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:46323 (local address 127.14.67.126:38019)
0901 20:01:26.403833 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.403836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.404514 (+ 678us) server_negotiation.cc:407] Connection header received
0901 20:01:26.405291 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.405295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.405354 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.405455 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:26.406708 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.407189 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.407998 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.408176 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.410545 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:26.410566 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:26.410569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:26.410592 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:26.412210 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.412754 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.412761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.412766 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.412825 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.413129 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.413132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.413134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.413289 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:26.413389 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.413961 (+ 572us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.414083 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20250901 20:01:26.414294 16584 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:46739
I20250901 20:01:26.414357 16714 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:46739 every 8 connection(s)
I20250901 20:01:26.414912 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.403999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38019 (local address 127.14.67.66:46323)
0901 20:01:26.404394 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.404428 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.405115 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.405590 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.405598 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.405964 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:26.406562 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.406580 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.407304 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.407308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.407891 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.407900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.408077 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.408699 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:26.408732 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:26.410384 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:26.412370 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.412376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.412386 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.412667 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.412925 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.412929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.412931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.413030 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.413406 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:26.413412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:26.413651 (+ 239us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.413874 (+ 223us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.414146 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":115,"threads_started":1}
I20250901 20:01:26.415321 16584 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:01:26.415660 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 16584
I20250901 20:01:26.415791 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:01:26.416064 14605 external_mini_cluster.cc:1468] Setting key 325412a72fe6f214e15d7d619098972a
I20250901 20:01:26.416684 16715 heartbeater.cc:344] Connected to a master server at 127.14.67.126:38019
I20250901 20:01:26.416833 16715 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:26.417374 16715 heartbeater.cc:507] Master 127.14.67.126:38019 requested a full tablet report, sending...
I20250901 20:01:26.418335 16363 ts_manager.cc:194] Registered new tserver with Master: 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739)
I20250901 20:01:26.419099 16363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:46323
I20250901 20:01:26.425089 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.418399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:48573 (local address 127.14.67.126:38019)
0901 20:01:26.418539 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.418545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.418713 (+ 168us) server_negotiation.cc:407] Connection header received
0901 20:01:26.418912 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.418915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.418957 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.419033 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:26.419647 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.420347 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.421289 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.421408 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.422384 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:26.422397 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:26.422399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:26.422420 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:26.423766 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.424195 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.424197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.424198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.424229 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.424462 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.424464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.424465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.424588 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:26.424655 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.424918 (+ 263us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.424992 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.14.67.67@KRBTEST.COM; defaulting to no policy
I20250901 20:01:26.425282 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.418333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38019 (local address 127.14.67.66:48573)
0901 20:01:26.418630 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.418642 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.418785 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.419059 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.419063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.419274 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:26.419530 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.419538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.420464 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.420513 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.421176 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.421190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.421306 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.421787 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:26.421808 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:26.422218 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:26.423873 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.423879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.423884 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.424105 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.424304 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.424307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.424309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.424371 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.424678 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:26.424684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:26.424776 (+ 92us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.424889 (+ 113us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.425002 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":80,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:26.457270 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:38019
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:26.555577 16726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:26.555748 16726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:26.555807 16726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:26.558897 16726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:26.558945 16726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:26.558996 16726 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20250901 20:01:26.559017 16726 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250901 20:01:26.559055 16726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:01:26.563004 16726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.67.126:38019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:26.563841 16726 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:26.564538 16726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:26.570429 16731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:26.570362 16732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:26.570353 16734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:26.570643 16726 server_base.cc:1047] running on GCE node
I20250901 20:01:26.571007 16726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:26.571493 16726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:26.572635 16726 hybrid_clock.cc:648] HybridClock initialized: now 1756756886572624 us; error 25 us; skew 500 ppm
Sep 01 20:01:26 dist-test-slave-2s4d krb5kdc[16320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756886, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:26.575073 16726 init.cc:377] Logged in from keytab as kudu/127.14.67.67@KRBTEST.COM (short username kudu)
I20250901 20:01:26.575964 16726 webserver.cc:480] Webserver started at http://127.14.67.67:36015/ using document root <none> and password file <none>
I20250901 20:01:26.576436 16726 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:26.576490 16726 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:26.576661 16726 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:26.578233 16726 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd"
format_stamp: "Formatted at 2025-09-01 20:01:26 on dist-test-slave-2s4d"
server_key: "8f9a1fd8a9ea7f8985682d29c056817c"
server_key_iv: "40634ed15eb6e76fafc47ea40a4c6546"
server_key_version: "encryptionkey@0"
I20250901 20:01:26.578720 16726 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd"
format_stamp: "Formatted at 2025-09-01 20:01:26 on dist-test-slave-2s4d"
server_key: "8f9a1fd8a9ea7f8985682d29c056817c"
server_key_iv: "40634ed15eb6e76fafc47ea40a4c6546"
server_key_version: "encryptionkey@0"
I20250901 20:01:26.581787 16726 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20250901 20:01:26.583823 16741 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:26.584669 16726 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:01:26.584781 16726 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd"
format_stamp: "Formatted at 2025-09-01 20:01:26 on dist-test-slave-2s4d"
server_key: "8f9a1fd8a9ea7f8985682d29c056817c"
server_key_iv: "40634ed15eb6e76fafc47ea40a4c6546"
server_key_version: "encryptionkey@0"
I20250901 20:01:26.584857 16726 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:26.611302 16726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:26.613708 16726 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:26.613857 16726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:26.615150 16726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:26.615237 16726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:26.615303 16726 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:26.615335 16726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 01 20:01:26 dist-test-slave-2s4d krb5kdc[16320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756886, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:26.626277 16726 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:39025
I20250901 20:01:26.626299 16856 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:39025 every 8 connection(s)
I20250901 20:01:26.627188 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.615668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:43157 (local address 127.14.67.126:38019)
0901 20:01:26.615810 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.615814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.616483 (+ 669us) server_negotiation.cc:407] Connection header received
0901 20:01:26.617301 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.617306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.617370 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.617511 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:26.619047 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.619641 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.620511 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.620691 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.623439 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:26.623463 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:26.623466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:26.623490 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:26.625225 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.625757 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.625759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.625761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.625799 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.626131 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.626135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.626137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.626341 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:26.626455 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.626910 (+ 455us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.627034 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250901 20:01:26.627314 16726 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:01:26.627696 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.615949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38019 (local address 127.14.67.67:43157)
0901 20:01:26.616341 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.616374 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.617085 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.617702 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.617720 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.618076 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:26.618842 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.618862 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.619770 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.619774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.620384 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.620395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.620593 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.621211 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:26.621245 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:26.623282 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:26.625340 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.625345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.625355 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.625660 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.625886 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.625893 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.625896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.626002 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.626446 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:26.626451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:26.626683 (+ 232us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.626845 (+ 162us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.627032 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":113,"threads_started":1}
I20250901 20:01:26.629072 16857 heartbeater.cc:344] Connected to a master server at 127.14.67.126:38019
I20250901 20:01:26.629215 16857 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:26.629753 16857 heartbeater.cc:507] Master 127.14.67.126:38019 requested a full tablet report, sending...
I20250901 20:01:26.630753 16368 ts_manager.cc:194] Registered new tserver with Master: 428f4b4d9c224b7bb0e9b1610fc0f8dd (127.14.67.67:39025)
I20250901 20:01:26.631369 16368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:43157
I20250901 20:01:26.631963 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 16726
I20250901 20:01:26.632076 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:01:26.632313 14605 external_mini_cluster.cc:1468] Setting key a5b035f283c055a3af420703ea7cab56
I20250901 20:01:26.634155 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250901 20:01:26.637143 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.631208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:57471 (local address 127.14.67.126:38019)
0901 20:01:26.631311 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.631315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.631551 (+ 236us) server_negotiation.cc:407] Connection header received
0901 20:01:26.631653 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.631656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.631702 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.631793 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:26.632393 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.632941 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.633587 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.633715 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.634550 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:26.634567 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:26.634569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:26.634589 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:26.635881 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.636238 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.636240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.636241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.636276 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.636478 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.636481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.636483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.636611 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:26.636696 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.636911 (+ 215us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.637002 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20250901 20:01:26.637151 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.631107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38019 (local address 127.14.67.67:57471)
0901 20:01:26.631464 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.631478 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.631570 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.631816 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.631819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.632006 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:26.632270 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.632278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.633050 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.633053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.633490 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.633500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.633607 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.634121 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:26.634138 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:26.634420 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:26.635982 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.635986 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.635988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.636143 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.636351 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.636356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.636358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.636399 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.636700 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:26.636704 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:26.636772 (+ 68us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.636857 (+ 85us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.636970 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":92,"threads_started":1}
Sep 01 20:01:26 dist-test-slave-2s4d krb5kdc[16320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 01 20:01:26 dist-test-slave-2s4d krb5kdc[16320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:26.655964 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.649340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.14.67.126:38019)
0901 20:01:26.649463 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.649466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.649477 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:01:26.649506 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.649508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.649543 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.649613 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:26.650300 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.650832 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.651394 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.651524 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.653443 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:26.653456 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:26.653458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:26.653479 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:26.654775 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.655108 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.655110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.655111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.655167 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.655328 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.655330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.655331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.655459 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:26.655554 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.655760 (+ 206us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.655847 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250901 20:01:26.658217 16368 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 20:01:26.660408 16368 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:01:26.671032 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.667136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:33477 (local address 127.0.0.1:53598)
0901 20:01:26.667402 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.667436 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.667566 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.668378 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.668383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.668403 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:26.668666 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.668676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.669744 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.669747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.670585 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.670592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.670691 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.670739 (+ 48us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.670823 (+ 84us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.670905 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20250901 20:01:26.671594 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.667266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.14.67.65:33477)
0901 20:01:26.667981 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.667986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.668011 (+ 25us) server_negotiation.cc:407] Connection header received
0901 20:01:26.668097 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.668105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.668251 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.668372 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:26.668798 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.669639 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.670701 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.671226 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.671334 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.671392 (+ 58us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.671479 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":65,"threads_started":1}
I20250901 20:01:26.671845 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.667493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:46739 (local address 127.0.0.1:39788)
0901 20:01:26.667717 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.667735 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.667814 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.668652 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.668655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.668674 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:26.668941 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.668949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.670398 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.670401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.671486 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.671496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.671599 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.671615 (+ 16us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.671665 (+ 50us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.671720 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20250901 20:01:26.672695 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.668228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.14.67.67:39025)
0901 20:01:26.668704 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.668709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.668729 (+ 20us) server_negotiation.cc:407] Connection header received
0901 20:01:26.668801 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.668806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.668936 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.669055 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:26.669577 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.670302 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.671794 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.672336 (+ 542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.672426 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.672495 (+ 69us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.672569 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":89,"threads_started":1}
I20250901 20:01:26.672945 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.667939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.14.67.66:46739)
0901 20:01:26.668206 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.668212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.668233 (+ 21us) server_negotiation.cc:407] Connection header received
0901 20:01:26.668311 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.668319 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.668493 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.669076 (+ 583us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:26.669249 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.670283 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.672095 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.672614 (+ 519us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.672710 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.672773 (+ 63us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.672848 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
I20250901 20:01:26.673152 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.667757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:39025 (local address 127.0.0.1:42686)
0901 20:01:26.667995 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.668007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.668073 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.669199 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.669201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.669221 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:26.669466 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.669473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.670408 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.670410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.671526 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.671535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.671611 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.671621 (+ 10us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.671665 (+ 44us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.671717 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20250901 20:01:26.674118 16507 tablet_service.cc:1468] Processing CreateTablet for tablet c7096004f1eb427db1136fa3896196fd (DEFAULT_TABLE table=test-table [id=b3181be4535743f3a5353a8d33643c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:26.675134 16507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7096004f1eb427db1136fa3896196fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:26.675108 16649 tablet_service.cc:1468] Processing CreateTablet for tablet c7096004f1eb427db1136fa3896196fd (DEFAULT_TABLE table=test-table [id=b3181be4535743f3a5353a8d33643c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:26.675967 16649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7096004f1eb427db1136fa3896196fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:26.681303 16881 tablet_bootstrap.cc:492] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb: Bootstrap starting.
I20250901 20:01:26.681303 16880 tablet_bootstrap.cc:492] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d: Bootstrap starting.
I20250901 20:01:26.683068 16791 tablet_service.cc:1468] Processing CreateTablet for tablet c7096004f1eb427db1136fa3896196fd (DEFAULT_TABLE table=test-table [id=b3181be4535743f3a5353a8d33643c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:26.683517 16880 tablet_bootstrap.cc:654] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:26.683722 16881 tablet_bootstrap.cc:654] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:26.683851 16791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7096004f1eb427db1136fa3896196fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:26.684409 16880 log.cc:826] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:26.684907 16881 log.cc:826] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:26.686679 16881 tablet_bootstrap.cc:492] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb: No bootstrap required, opened a new log
I20250901 20:01:26.686679 16880 tablet_bootstrap.cc:492] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d: No bootstrap required, opened a new log
I20250901 20:01:26.686858 16880 ts_tablet_manager.cc:1397] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250901 20:01:26.686887 16881 ts_tablet_manager.cc:1397] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250901 20:01:26.689214 16884 tablet_bootstrap.cc:492] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Bootstrap starting.
I20250901 20:01:26.689831 16881 raft_consensus.cc:357] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } }
I20250901 20:01:26.689831 16880 raft_consensus.cc:357] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } }
I20250901 20:01:26.690049 16880 raft_consensus.cc:383] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:26.690049 16881 raft_consensus.cc:383] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:26.690109 16881 raft_consensus.cc:738] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ed5d6178c447ce820fd6a9bd5b6ecb, State: Initialized, Role: FOLLOWER
I20250901 20:01:26.690132 16880 raft_consensus.cc:738] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca3083eb2e1425a828f5f42e283d36d, State: Initialized, Role: FOLLOWER
I20250901 20:01:26.690580 16881 consensus_queue.cc:260] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } }
I20250901 20:01:26.690580 16880 consensus_queue.cc:260] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } }
I20250901 20:01:26.690970 16884 tablet_bootstrap.cc:654] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:26.691408 16880 ts_tablet_manager.cc:1428] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250901 20:01:26.691408 16881 ts_tablet_manager.cc:1428] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250901 20:01:26.691761 16715 heartbeater.cc:499] Master 127.14.67.126:38019 was elected leader, sending a full tablet report...
I20250901 20:01:26.691920 16573 heartbeater.cc:499] Master 127.14.67.126:38019 was elected leader, sending a full tablet report...
I20250901 20:01:26.691958 16884 log.cc:826] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:26.693452 16884 tablet_bootstrap.cc:492] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd: No bootstrap required, opened a new log
I20250901 20:01:26.693607 16884 ts_tablet_manager.cc:1397] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250901 20:01:26.695930 16884 raft_consensus.cc:357] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } }
I20250901 20:01:26.696082 16884 raft_consensus.cc:383] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:26.696139 16884 raft_consensus.cc:738] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428f4b4d9c224b7bb0e9b1610fc0f8dd, State: Initialized, Role: FOLLOWER
I20250901 20:01:26.696460 16884 consensus_queue.cc:260] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } }
I20250901 20:01:26.696983 16857 heartbeater.cc:499] Master 127.14.67.126:38019 was elected leader, sending a full tablet report...
I20250901 20:01:26.697134 16884 ts_tablet_manager.cc:1428] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20250901 20:01:26.707139 16574 tablet.cc:2378] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 20:01:26.784688 16888 raft_consensus.cc:491] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:01:26.784890 16888 raft_consensus.cc:513] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } }
I20250901 20:01:26.785940 16888 leader_election.cc:290] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477), 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739)
I20250901 20:01:26.788997 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.786139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:33477 (local address 127.14.67.67:41725)
0901 20:01:26.786285 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.786302 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.786404 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.786692 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.786696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.786721 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:26.786976 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.786983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.787836 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.787838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.788607 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.788616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.788733 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.788752 (+ 19us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.788797 (+ 45us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.788850 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20250901 20:01:26.789234 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.786139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:46739 (local address 127.14.67.67:47163)
0901 20:01:26.786678 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.786691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.786769 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.786973 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.786976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.786994 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:26.787233 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.787238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.788063 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.788066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.788868 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.788875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.788965 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.788978 (+ 13us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.789025 (+ 47us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.789079 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":217,"thread_start_us":135,"threads_started":1}
I20250901 20:01:26.789465 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.786239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:41725 (local address 127.14.67.65:33477)
0901 20:01:26.786383 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.786389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.786405 (+ 16us) server_negotiation.cc:407] Connection header received
0901 20:01:26.786528 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.786532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.786585 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.786661 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:26.787082 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.787733 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.788723 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.789189 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.789226 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.789284 (+ 58us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.789336 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250901 20:01:26.789638 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.786308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:47163 (local address 127.14.67.66:46739)
0901 20:01:26.786448 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.786455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.786764 (+ 309us) server_negotiation.cc:407] Connection header received
0901 20:01:26.786851 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.786853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.786891 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.786950 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:26.787332 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.787979 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.788982 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.789404 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.789433 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.789476 (+ 43us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.789532 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250901 20:01:26.790047 16527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7096004f1eb427db1136fa3896196fd" candidate_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca3083eb2e1425a828f5f42e283d36d" is_pre_election: true
I20250901 20:01:26.790132 16669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7096004f1eb427db1136fa3896196fd" candidate_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" is_pre_election: true
I20250901 20:01:26.790292 16527 raft_consensus.cc:2466] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428f4b4d9c224b7bb0e9b1610fc0f8dd in term 0.
I20250901 20:01:26.790361 16669 raft_consensus.cc:2466] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428f4b4d9c224b7bb0e9b1610fc0f8dd in term 0.
I20250901 20:01:26.790746 16742 leader_election.cc:304] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428f4b4d9c224b7bb0e9b1610fc0f8dd, 6ca3083eb2e1425a828f5f42e283d36d; no voters:
I20250901 20:01:26.790963 16888 raft_consensus.cc:2802] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:01:26.791029 16888 raft_consensus.cc:491] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:01:26.791060 16888 raft_consensus.cc:3058] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:26.791951 16888 raft_consensus.cc:513] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } }
I20250901 20:01:26.792249 16888 leader_election.cc:290] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [CANDIDATE]: Term 1 election: Requested vote from peers 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477), 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739)
I20250901 20:01:26.792526 16527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7096004f1eb427db1136fa3896196fd" candidate_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca3083eb2e1425a828f5f42e283d36d"
I20250901 20:01:26.792550 16669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7096004f1eb427db1136fa3896196fd" candidate_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb"
I20250901 20:01:26.792622 16527 raft_consensus.cc:3058] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:26.792650 16669 raft_consensus.cc:3058] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:26.793463 16527 raft_consensus.cc:2466] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428f4b4d9c224b7bb0e9b1610fc0f8dd in term 1.
I20250901 20:01:26.793571 16669 raft_consensus.cc:2466] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428f4b4d9c224b7bb0e9b1610fc0f8dd in term 1.
I20250901 20:01:26.793736 16742 leader_election.cc:304] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428f4b4d9c224b7bb0e9b1610fc0f8dd, 6ca3083eb2e1425a828f5f42e283d36d; no voters:
I20250901 20:01:26.793911 16888 raft_consensus.cc:2802] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:26.794111 16888 raft_consensus.cc:695] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 LEADER]: Becoming Leader. State: Replica: 428f4b4d9c224b7bb0e9b1610fc0f8dd, State: Running, Role: LEADER
I20250901 20:01:26.794382 16888 consensus_queue.cc:237] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } }
I20250901 20:01:26.797081 16368 catalog_manager.cc:5582] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 428f4b4d9c224b7bb0e9b1610fc0f8dd (127.14.67.67). New cstate: current_term: 1 leader_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } health_report { overall_health: UNKNOWN } } }
Sep 01 20:01:26 dist-test-slave-2s4d krb5kdc[16320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756885, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:26.843156 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.836107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38019 (local address 127.0.0.1:35484)
0901 20:01:26.836370 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.836382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.836486 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.836734 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.836737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.837155 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:26.837340 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.837346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.838016 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.838019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.838376 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.838382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.838492 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.839095 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:26.839107 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:26.840401 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:26.841813 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.841821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.841826 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.842027 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.842224 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:26.842226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:26.842228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:26.842268 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:26.842601 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:26.842609 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:26.842700 (+ 91us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.842809 (+ 109us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.842956 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
I20250901 20:01:26.843156 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.836187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.14.67.126:38019)
0901 20:01:26.836323 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.836328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.836448 (+ 120us) server_negotiation.cc:407] Connection header received
0901 20:01:26.836569 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.836574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.836611 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.836678 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:26.837448 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.837915 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.838478 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.838685 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.840507 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:26.840522 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:26.840524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:26.840544 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:26.841710 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.842111 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.842114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.842116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.842150 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:26.842339 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:26.842341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:26.842343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:26.842479 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:26.842601 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.842858 (+ 257us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.843019 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":44}
I20250901 20:01:26.846412 16368 catalog_manager.cc:2232] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:35484:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250901 20:01:26.847070 16368 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:01:26.851302 16507 tablet_service.cc:1468] Processing CreateTablet for tablet 4806dde239984238bcb2ddcc516cb1d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b182865b7464737a29d58083049e803]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250901 20:01:26.851311 16649 tablet_service.cc:1468] Processing CreateTablet for tablet 4806dde239984238bcb2ddcc516cb1d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b182865b7464737a29d58083049e803]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250901 20:01:26.851917 16649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4806dde239984238bcb2ddcc516cb1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:26.851918 16507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4806dde239984238bcb2ddcc516cb1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:26.851598 16791 tablet_service.cc:1468] Processing CreateTablet for tablet 4806dde239984238bcb2ddcc516cb1d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1b182865b7464737a29d58083049e803]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250901 20:01:26.852144 16791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4806dde239984238bcb2ddcc516cb1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:26.854332 16884 tablet_bootstrap.cc:492] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Bootstrap starting.
I20250901 20:01:26.854602 16881 tablet_bootstrap.cc:492] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Bootstrap starting.
I20250901 20:01:26.854619 16880 tablet_bootstrap.cc:492] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d: Bootstrap starting.
I20250901 20:01:26.855373 16884 tablet_bootstrap.cc:654] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:26.855460 16881 tablet_bootstrap.cc:654] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:26.855620 16880 tablet_bootstrap.cc:654] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:26.856592 16881 tablet_bootstrap.cc:492] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb: No bootstrap required, opened a new log
I20250901 20:01:26.856684 16881 ts_tablet_manager.cc:1397] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:26.857163 16881 raft_consensus.cc:357] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:26.857249 16884 tablet_bootstrap.cc:492] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: No bootstrap required, opened a new log
I20250901 20:01:26.857263 16881 raft_consensus.cc:383] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:26.857301 16881 raft_consensus.cc:738] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ed5d6178c447ce820fd6a9bd5b6ecb, State: Initialized, Role: FOLLOWER
I20250901 20:01:26.857328 16884 ts_tablet_manager.cc:1397] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250901 20:01:26.857431 16881 consensus_queue.cc:260] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:26.857682 16881 ts_tablet_manager.cc:1428] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:01:26.857708 16880 tablet_bootstrap.cc:492] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d: No bootstrap required, opened a new log
I20250901 20:01:26.857782 16880 ts_tablet_manager.cc:1397] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250901 20:01:26.857650 16885 tablet_replica.cc:440] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } }
I20250901 20:01:26.857784 16884 raft_consensus.cc:357] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:26.857864 16885 tablet_replica.cc:443] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:26.857873 16884 raft_consensus.cc:383] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:26.857911 16884 raft_consensus.cc:738] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428f4b4d9c224b7bb0e9b1610fc0f8dd, State: Initialized, Role: FOLLOWER
I20250901 20:01:26.858019 16884 consensus_queue.cc:260] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:26.858250 16884 ts_tablet_manager.cc:1428] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:01:26.858220 16880 raft_consensus.cc:357] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:26.858309 16880 raft_consensus.cc:383] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:26.858341 16880 raft_consensus.cc:738] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca3083eb2e1425a828f5f42e283d36d, State: Initialized, Role: FOLLOWER
I20250901 20:01:26.858494 16880 consensus_queue.cc:260] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:26.858749 16880 ts_tablet_manager.cc:1428] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250901 20:01:26.858858 16886 tablet_replica.cc:440] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } }
I20250901 20:01:26.858958 16886 tablet_replica.cc:443] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:26.858989 16888 tablet_replica.cc:440] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } }
I20250901 20:01:26.859076 16888 tablet_replica.cc:443] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: This TxnStatusTablet replica's current role is: FOLLOWER
W20250901 20:01:26.878196 16858 tablet.cc:2378] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250901 20:01:26.916821 16716 tablet.cc:2378] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 20:01:26.951711 16888 raft_consensus.cc:491] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:01:26.951829 16888 raft_consensus.cc:513] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:26.952178 16888 leader_election.cc:290] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477), 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739)
I20250901 20:01:26.952597 16527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4806dde239984238bcb2ddcc516cb1d9" candidate_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca3083eb2e1425a828f5f42e283d36d" is_pre_election: true
I20250901 20:01:26.952597 16669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4806dde239984238bcb2ddcc516cb1d9" candidate_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" is_pre_election: true
I20250901 20:01:26.952744 16669 raft_consensus.cc:2466] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428f4b4d9c224b7bb0e9b1610fc0f8dd in term 0.
I20250901 20:01:26.952744 16527 raft_consensus.cc:2466] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428f4b4d9c224b7bb0e9b1610fc0f8dd in term 0.
I20250901 20:01:26.953049 16745 leader_election.cc:304] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ed5d6178c447ce820fd6a9bd5b6ecb, 428f4b4d9c224b7bb0e9b1610fc0f8dd; no voters:
I20250901 20:01:26.953217 16888 raft_consensus.cc:2802] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:01:26.953270 16888 raft_consensus.cc:491] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:01:26.953298 16888 raft_consensus.cc:3058] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:26.954164 16888 raft_consensus.cc:513] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:26.954456 16888 leader_election.cc:290] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [CANDIDATE]: Term 1 election: Requested vote from peers 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477), 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739)
I20250901 20:01:26.954795 16527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4806dde239984238bcb2ddcc516cb1d9" candidate_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca3083eb2e1425a828f5f42e283d36d"
I20250901 20:01:26.954816 16669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4806dde239984238bcb2ddcc516cb1d9" candidate_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb"
I20250901 20:01:26.954905 16527 raft_consensus.cc:3058] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:26.954905 16669 raft_consensus.cc:3058] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:26.955904 16669 raft_consensus.cc:2466] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428f4b4d9c224b7bb0e9b1610fc0f8dd in term 1.
I20250901 20:01:26.955904 16527 raft_consensus.cc:2466] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428f4b4d9c224b7bb0e9b1610fc0f8dd in term 1.
I20250901 20:01:26.956161 16745 leader_election.cc:304] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ed5d6178c447ce820fd6a9bd5b6ecb, 428f4b4d9c224b7bb0e9b1610fc0f8dd; no voters:
I20250901 20:01:26.956339 16888 raft_consensus.cc:2802] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:26.956439 16888 raft_consensus.cc:695] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 LEADER]: Becoming Leader. State: Replica: 428f4b4d9c224b7bb0e9b1610fc0f8dd, State: Running, Role: LEADER
I20250901 20:01:26.956576 16888 consensus_queue.cc:237] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:26.957195 16890 tablet_replica.cc:440] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: TxnStatusTablet state changed. Reason: New leader 428f4b4d9c224b7bb0e9b1610fc0f8dd. Latest consensus state: current_term: 1 leader_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } }
I20250901 20:01:26.957301 16890 tablet_replica.cc:443] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: This TxnStatusTablet replica's current role is: LEADER
I20250901 20:01:26.957571 16902 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250901 20:01:26.957795 16902 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250901 20:01:26.958103 16367 catalog_manager.cc:5582] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 428f4b4d9c224b7bb0e9b1610fc0f8dd (127.14.67.67). New cstate: current_term: 1 leader_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } health_report { overall_health: HEALTHY } } }
I20250901 20:01:26.977720 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.974840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.14.67.67:39025)
0901 20:01:26.974982 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.974986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.974999 (+ 13us) server_negotiation.cc:407] Connection header received
0901 20:01:26.975142 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.975145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.975196 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.975269 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:26.975597 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.976080 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.976677 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.976886 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.976951 (+ 65us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:26.977331 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:26.977452 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.977541 (+ 89us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.977588 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250901 20:01:26.977713 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.974763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:39025 (local address 127.0.0.1:42694)
0901 20:01:26.974902 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:26.974919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:26.975027 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:26.975283 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:26.975286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:26.975294 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0901 20:01:26.975498 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.975504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.976177 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.976180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:26.976579 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:26.976586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.976683 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.976724 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0901 20:01:26.977432 (+ 708us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0901 20:01:26.977453 (+ 21us) client_negotiation.cc:769] Sending connection context
0901 20:01:26.977515 (+ 62us) client_negotiation.cc:240] Negotiation successful
0901 20:01:26.977589 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20250901 20:01:26.981725 16669 raft_consensus.cc:1273] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Refusing update from remote peer 428f4b4d9c224b7bb0e9b1610fc0f8dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:26.981829 16527 raft_consensus.cc:1273] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Refusing update from remote peer 428f4b4d9c224b7bb0e9b1610fc0f8dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:26.982249 16890 consensus_queue.cc:1035] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:26.982380 16888 consensus_queue.cc:1035] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:26.986438 16885 tablet_replica.cc:440] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb: TxnStatusTablet state changed. Reason: New leader 428f4b4d9c224b7bb0e9b1610fc0f8dd. Latest consensus state: current_term: 1 leader_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } }
I20250901 20:01:26.986559 16885 tablet_replica.cc:443] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:26.987643 16905 mvcc.cc:204] Tried to move back new op lower bound from 7195676209071755264 to 7195676208975179776. Current Snapshot: MvccSnapshot[applied={T|T < 7195676209071755264}]
I20250901 20:01:26.987780 16890 tablet_replica.cc:440] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } }
I20250901 20:01:26.987857 16890 tablet_replica.cc:443] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: This TxnStatusTablet replica's current role is: LEADER
I20250901 20:01:26.988354 16886 tablet_replica.cc:440] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d: TxnStatusTablet state changed. Reason: New leader 428f4b4d9c224b7bb0e9b1610fc0f8dd. Latest consensus state: current_term: 1 leader_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } }
I20250901 20:01:26.988466 16886 tablet_replica.cc:443] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:26.990124 16888 tablet_replica.cc:440] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } }
I20250901 20:01:26.990211 16888 tablet_replica.cc:443] T 4806dde239984238bcb2ddcc516cb1d9 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: This TxnStatusTablet replica's current role is: LEADER
I20250901 20:01:26.992419 16885 tablet_replica.cc:440] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } }
I20250901 20:01:26.992518 16885 tablet_replica.cc:443] T 4806dde239984238bcb2ddcc516cb1d9 P 14ed5d6178c447ce820fd6a9bd5b6ecb: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:26.993817 16886 tablet_replica.cc:440] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } } }
I20250901 20:01:26.993904 16886 tablet_replica.cc:443] T 4806dde239984238bcb2ddcc516cb1d9 P 6ca3083eb2e1425a828f5f42e283d36d: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:01:26.994473 16906 mvcc.cc:204] Tried to move back new op lower bound from 7195676209071755264 to 7195676208975179776. Current Snapshot: MvccSnapshot[applied={T|T < 7195676209071755264}]
I20250901 20:01:26.994202 16907 mvcc.cc:204] Tried to move back new op lower bound from 7195676209071755264 to 7195676208975179776. Current Snapshot: MvccSnapshot[applied={T|T < 7195676209071755264 or (T in {7195676209071755264})}]
I20250901 20:01:26.997680 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:26.995186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.14.67.67:39025)
0901 20:01:26.995356 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:26.995361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:26.995374 (+ 13us) server_negotiation.cc:407] Connection header received
0901 20:01:26.995531 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:26.995534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:26.995571 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:26.995639 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:26.995976 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.996489 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:26.997103 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:26.997264 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:26.997301 (+ 37us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:26.997357 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:26.997419 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:26.997531 (+ 112us) server_negotiation.cc:299] Negotiation successful
0901 20:01:26.997569 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20250901 20:01:27.006456 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:27.003565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:39025 (local address 127.14.67.67:58389)
0901 20:01:27.003692 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:27.003706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:27.003781 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:27.003985 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:27.003987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:27.003994 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0901 20:01:27.004481 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:27.004486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.005387 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:27.005389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:27.005788 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:27.005794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.005870 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:27.005897 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0901 20:01:27.006234 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0901 20:01:27.006237 (+ 3us) client_negotiation.cc:769] Sending connection context
0901 20:01:27.006280 (+ 43us) client_negotiation.cc:240] Negotiation successful
0901 20:01:27.006325 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20250901 20:01:27.006477 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:27.003623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:58389 (local address 127.14.67.67:39025)
0901 20:01:27.003719 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:27.003725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:27.003752 (+ 27us) server_negotiation.cc:407] Connection header received
0901 20:01:27.003863 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:27.003866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:27.003905 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:27.003971 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:27.004579 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.005296 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:27.005872 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.006048 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:27.006091 (+ 43us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:27.006158 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:27.006234 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:27.006303 (+ 69us) server_negotiation.cc:299] Negotiation successful
0901 20:01:27.006343 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20250901 20:01:27.012120 16669 raft_consensus.cc:1273] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Refusing update from remote peer 428f4b4d9c224b7bb0e9b1610fc0f8dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:27.012434 16527 raft_consensus.cc:1273] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Refusing update from remote peer 428f4b4d9c224b7bb0e9b1610fc0f8dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:27.012511 16890 consensus_queue.cc:1035] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:27.012769 16890 consensus_queue.cc:1035] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:27.048645 16367 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35472:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:27.048815 16367 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35472:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:27.050976 16367 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de: Sending DeleteTablet for 3 replicas of tablet c7096004f1eb427db1136fa3896196fd
I20250901 20:01:27.051560 16507 tablet_service.cc:1515] Processing DeleteTablet for tablet c7096004f1eb427db1136fa3896196fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:27 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:53598
I20250901 20:01:27.051581 16790 tablet_service.cc:1515] Processing DeleteTablet for tablet c7096004f1eb427db1136fa3896196fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:27 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:42686
I20250901 20:01:27.051808 16649 tablet_service.cc:1515] Processing DeleteTablet for tablet c7096004f1eb427db1136fa3896196fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:27 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:39788
I20250901 20:01:27.052192 16921 tablet_replica.cc:331] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd: stopping tablet replica
I20250901 20:01:27.052248 16920 tablet_replica.cc:331] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d: stopping tablet replica
I20250901 20:01:27.052541 16920 raft_consensus.cc:2241] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:27.052583 16921 raft_consensus.cc:2241] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:01:27.052667 16367 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 20:01:27.052809 16920 raft_consensus.cc:2270] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:27.053058 16921 raft_consensus.cc:2270] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 20:01:27.053156 16367 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:01:27.054018 16920 ts_tablet_manager.cc:1905] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:27.054591 16921 ts_tablet_manager.cc:1905] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:27.057524 16920 ts_tablet_manager.cc:1918] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:01:27.057669 16920 log.cc:1199] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/wal/wals/c7096004f1eb427db1136fa3896196fd
I20250901 20:01:27.057803 16649 tablet_service.cc:1468] Processing CreateTablet for tablet 9a14419363fd4ce798c8c96eeed03366 (DEFAULT_TABLE table=test-table [id=8bb00436df944e4493b36057392107b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:27.058055 16649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a14419363fd4ce798c8c96eeed03366. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:27.058264 16921 ts_tablet_manager.cc:1918] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:01:27.058336 16921 log.cc:1199] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/c7096004f1eb427db1136fa3896196fd
I20250901 20:01:27.058377 16920 ts_tablet_manager.cc:1939] T c7096004f1eb427db1136fa3896196fd P 6ca3083eb2e1425a828f5f42e283d36d: Deleting consensus metadata
I20250901 20:01:27.058663 16921 ts_tablet_manager.cc:1939] T c7096004f1eb427db1136fa3896196fd P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Deleting consensus metadata
I20250901 20:01:27.059422 16790 tablet_service.cc:1468] Processing CreateTablet for tablet 9a14419363fd4ce798c8c96eeed03366 (DEFAULT_TABLE table=test-table [id=8bb00436df944e4493b36057392107b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:27.059444 16507 tablet_service.cc:1468] Processing CreateTablet for tablet 9a14419363fd4ce798c8c96eeed03366 (DEFAULT_TABLE table=test-table [id=8bb00436df944e4493b36057392107b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:27.059641 16790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a14419363fd4ce798c8c96eeed03366. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:27.059655 16507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a14419363fd4ce798c8c96eeed03366. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:27.061527 16880 tablet_bootstrap.cc:492] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d: Bootstrap starting.
I20250901 20:01:27.061597 16881 tablet_bootstrap.cc:492] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Bootstrap starting.
I20250901 20:01:27.061690 16884 tablet_bootstrap.cc:492] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Bootstrap starting.
I20250901 20:01:27.062485 16880 tablet_bootstrap.cc:654] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:27.062603 16884 tablet_bootstrap.cc:654] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:27.062812 16881 tablet_bootstrap.cc:654] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:27.063679 16352 catalog_manager.cc:4928] TS 428f4b4d9c224b7bb0e9b1610fc0f8dd (127.14.67.67:39025): tablet c7096004f1eb427db1136fa3896196fd (table test-table [id=b3181be4535743f3a5353a8d33643c75]) successfully deleted
I20250901 20:01:27.063877 16922 tablet_replica.cc:331] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb: stopping tablet replica
I20250901 20:01:27.064004 16352 catalog_manager.cc:4928] TS 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477): tablet c7096004f1eb427db1136fa3896196fd (table test-table [id=b3181be4535743f3a5353a8d33643c75]) successfully deleted
I20250901 20:01:27.064162 16922 raft_consensus.cc:2241] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:27.064337 16922 raft_consensus.cc:2270] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:27.065387 16884 tablet_bootstrap.cc:492] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: No bootstrap required, opened a new log
I20250901 20:01:27.065465 16884 ts_tablet_manager.cc:1397] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250901 20:01:27.065491 16922 ts_tablet_manager.cc:1905] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:27.065887 16880 tablet_bootstrap.cc:492] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d: No bootstrap required, opened a new log
I20250901 20:01:27.065869 16884 raft_consensus.cc:357] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:27.065959 16884 raft_consensus.cc:383] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:27.065961 16880 ts_tablet_manager.cc:1397] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20250901 20:01:27.065991 16884 raft_consensus.cc:738] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428f4b4d9c224b7bb0e9b1610fc0f8dd, State: Initialized, Role: FOLLOWER
I20250901 20:01:27.066109 16884 consensus_queue.cc:260] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:27.066327 16884 ts_tablet_manager.cc:1428] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250901 20:01:27.066398 16880 raft_consensus.cc:357] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:27.066530 16880 raft_consensus.cc:383] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:27.066581 16880 raft_consensus.cc:738] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca3083eb2e1425a828f5f42e283d36d, State: Initialized, Role: FOLLOWER
I20250901 20:01:27.066690 16880 consensus_queue.cc:260] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:27.066936 16880 ts_tablet_manager.cc:1428] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:01:27.067569 16881 tablet_bootstrap.cc:492] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb: No bootstrap required, opened a new log
I20250901 20:01:27.067641 16881 ts_tablet_manager.cc:1397] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20250901 20:01:27.068053 16881 raft_consensus.cc:357] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:27.068137 16881 raft_consensus.cc:383] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:27.068166 16881 raft_consensus.cc:738] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ed5d6178c447ce820fd6a9bd5b6ecb, State: Initialized, Role: FOLLOWER
I20250901 20:01:27.068261 16881 consensus_queue.cc:260] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:27.068485 16881 ts_tablet_manager.cc:1428] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250901 20:01:27.068655 16922 ts_tablet_manager.cc:1918] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:01:27.068727 16922 log.cc:1199] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/c7096004f1eb427db1136fa3896196fd
I20250901 20:01:27.069067 16922 ts_tablet_manager.cc:1939] T c7096004f1eb427db1136fa3896196fd P 14ed5d6178c447ce820fd6a9bd5b6ecb: Deleting consensus metadata
I20250901 20:01:27.070228 16355 catalog_manager.cc:4928] TS 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739): tablet c7096004f1eb427db1136fa3896196fd (table test-table [id=b3181be4535743f3a5353a8d33643c75]) successfully deleted
I20250901 20:01:27.134924 16886 raft_consensus.cc:491] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:01:27.135048 16886 raft_consensus.cc:513] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:27.135998 16886 leader_election.cc:290] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739), 428f4b4d9c224b7bb0e9b1610fc0f8dd (127.14.67.67:39025)
I20250901 20:01:27.139118 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:27.136189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:46739 (local address 127.14.67.65:42931)
0901 20:01:27.136496 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:27.136513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:27.136661 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:27.137075 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:27.137077 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:27.137124 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:27.137392 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:27.137401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.138120 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:27.138123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:27.138728 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:27.138735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.138837 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:27.138860 (+ 23us) client_negotiation.cc:769] Sending connection context
0901 20:01:27.138911 (+ 51us) client_negotiation.cc:240] Negotiation successful
0901 20:01:27.138967 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"spinlock_wait_cycles":3200,"thread_start_us":76,"threads_started":1}
I20250901 20:01:27.139559 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:27.136397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:42931 (local address 127.14.67.66:46739)
0901 20:01:27.136527 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:27.136532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:27.136580 (+ 48us) server_negotiation.cc:407] Connection header received
0901 20:01:27.136809 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:27.136813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:27.136880 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:27.136933 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:27.137524 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.138011 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:27.138837 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.139292 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:27.139336 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:27.139384 (+ 48us) server_negotiation.cc:299] Negotiation successful
0901 20:01:27.139455 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20250901 20:01:27.139912 16669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a14419363fd4ce798c8c96eeed03366" candidate_uuid: "6ca3083eb2e1425a828f5f42e283d36d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" is_pre_election: true
I20250901 20:01:27.140045 16669 raft_consensus.cc:2466] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ca3083eb2e1425a828f5f42e283d36d in term 0.
I20250901 20:01:27.140395 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:27.136232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:39025 (local address 127.14.67.65:57475)
0901 20:01:27.136591 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:27.136604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:27.136662 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:27.137069 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:27.137072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:27.137111 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:27.137397 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:27.137403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.138373 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:27.138376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:27.139344 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:27.139355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.140095 (+ 740us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:27.140110 (+ 15us) client_negotiation.cc:769] Sending connection context
0901 20:01:27.140209 (+ 99us) client_negotiation.cc:240] Negotiation successful
0901 20:01:27.140261 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"spinlock_wait_cycles":3840,"thread_start_us":97,"threads_started":1}
I20250901 20:01:27.140470 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:27.136632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:57475 (local address 127.14.67.67:39025)
0901 20:01:27.136748 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:27.136752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:27.136777 (+ 25us) server_negotiation.cc:407] Connection header received
0901 20:01:27.136826 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:27.136829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:27.136892 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:27.136966 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:27.137520 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.138252 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:27.139510 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.140003 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:27.140037 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:27.140265 (+ 228us) server_negotiation.cc:299] Negotiation successful
0901 20:01:27.140336 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250901 20:01:27.140389 16461 leader_election.cc:304] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ed5d6178c447ce820fd6a9bd5b6ecb, 6ca3083eb2e1425a828f5f42e283d36d; no voters:
I20250901 20:01:27.140810 16886 raft_consensus.cc:2802] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:01:27.140872 16886 raft_consensus.cc:491] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:01:27.140911 16886 raft_consensus.cc:3058] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:27.140985 16811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a14419363fd4ce798c8c96eeed03366" candidate_uuid: "6ca3083eb2e1425a828f5f42e283d36d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" is_pre_election: true
I20250901 20:01:27.141157 16811 raft_consensus.cc:2466] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ca3083eb2e1425a828f5f42e283d36d in term 0.
I20250901 20:01:27.141973 16886 raft_consensus.cc:513] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:27.142374 16886 leader_election.cc:290] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [CANDIDATE]: Term 1 election: Requested vote from peers 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739), 428f4b4d9c224b7bb0e9b1610fc0f8dd (127.14.67.67:39025)
I20250901 20:01:27.142714 16669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a14419363fd4ce798c8c96eeed03366" candidate_uuid: "6ca3083eb2e1425a828f5f42e283d36d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb"
I20250901 20:01:27.142828 16669 raft_consensus.cc:3058] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:27.143810 16669 raft_consensus.cc:2466] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ca3083eb2e1425a828f5f42e283d36d in term 1.
I20250901 20:01:27.144022 16811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a14419363fd4ce798c8c96eeed03366" candidate_uuid: "6ca3083eb2e1425a828f5f42e283d36d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd"
I20250901 20:01:27.144062 16461 leader_election.cc:304] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ed5d6178c447ce820fd6a9bd5b6ecb, 6ca3083eb2e1425a828f5f42e283d36d; no voters:
I20250901 20:01:27.144124 16811 raft_consensus.cc:3058] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:27.144212 16886 raft_consensus.cc:2802] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:27.144368 16886 raft_consensus.cc:695] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 1 LEADER]: Becoming Leader. State: Replica: 6ca3083eb2e1425a828f5f42e283d36d, State: Running, Role: LEADER
I20250901 20:01:27.144593 16886 consensus_queue.cc:237] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:27.145052 16811 raft_consensus.cc:2466] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ca3083eb2e1425a828f5f42e283d36d in term 1.
I20250901 20:01:27.146560 16363 catalog_manager.cc:5582] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65). New cstate: current_term: 1 leader_uuid: "6ca3083eb2e1425a828f5f42e283d36d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20250901 20:01:27.167611 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:27.164470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.14.67.65:33477)
0901 20:01:27.164583 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:27.164587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:27.164601 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:01:27.164738 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:27.164741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:27.164802 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:27.164901 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:27.165148 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.165882 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:27.166434 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.166642 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:27.166708 (+ 66us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:27.167255 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:27.167367 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:27.167439 (+ 72us) server_negotiation.cc:299] Negotiation successful
0901 20:01:27.167489 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250901 20:01:27.176294 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:27.173751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:36579 (local address 127.14.67.67:39025)
0901 20:01:27.173869 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:27.173876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:27.174050 (+ 174us) server_negotiation.cc:407] Connection header received
0901 20:01:27.174165 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:27.174168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:27.174206 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:27.174263 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:27.174667 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.175094 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:27.175705 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.175881 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:27.175920 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:27.175974 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:27.176037 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:27.176134 (+ 97us) server_negotiation.cc:299] Negotiation successful
0901 20:01:27.176173 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250901 20:01:27.176316 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:27.173706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:39025 (local address 127.14.67.65:36579)
0901 20:01:27.173982 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:27.173993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:27.174082 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:27.174307 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:27.174310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:27.174319 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0901 20:01:27.174566 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:27.174573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.175184 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:27.175187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:27.175606 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:27.175612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.175719 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:27.175754 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0901 20:01:27.176051 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0901 20:01:27.176057 (+ 6us) client_negotiation.cc:769] Sending connection context
0901 20:01:27.176109 (+ 52us) client_negotiation.cc:240] Negotiation successful
0901 20:01:27.176173 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":73,"threads_started":1}
I20250901 20:01:27.181879 16669 raft_consensus.cc:1273] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Refusing update from remote peer 6ca3083eb2e1425a828f5f42e283d36d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:27.181944 16811 raft_consensus.cc:1273] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Refusing update from remote peer 6ca3083eb2e1425a828f5f42e283d36d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:27.182307 16928 consensus_queue.cc:1035] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:27.182477 16886 consensus_queue.cc:1035] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [LEADER]: Connected to new peer: Peer: permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:27.199038 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:27.196614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:50097 (local address 127.14.67.65:33477)
0901 20:01:27.196717 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:27.196723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:27.196737 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:01:27.196778 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:27.196783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:27.196822 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:27.196885 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:27.197210 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.197806 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:27.198382 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.198592 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:27.198643 (+ 51us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:27.198717 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:27.198784 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:27.198887 (+ 103us) server_negotiation.cc:299] Negotiation successful
0901 20:01:27.198926 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20250901 20:01:27.199038 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:27.196380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:33477 (local address 127.14.67.67:50097)
0901 20:01:27.196600 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:27.196614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:27.196686 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:27.196898 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:27.196901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:27.196908 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0901 20:01:27.197106 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:27.197113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.197902 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:27.197905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:27.198282 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:27.198288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:27.198383 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:27.198408 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0901 20:01:27.198801 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0901 20:01:27.198807 (+ 6us) client_negotiation.cc:769] Sending connection context
0901 20:01:27.198860 (+ 53us) client_negotiation.cc:240] Negotiation successful
0901 20:01:27.198926 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
W20250901 20:01:27.199831 16907 tablet_replica.cc:1304] Aborted: operation has been aborted: cancelling pending write operations
I20250901 20:01:28.765473 16363 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35472:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:28.765605 16363 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35472:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:28.767752 16363 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de: Sending DeleteTablet for 3 replicas of tablet 9a14419363fd4ce798c8c96eeed03366
I20250901 20:01:28.768225 16507 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a14419363fd4ce798c8c96eeed03366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:28 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:53598
I20250901 20:01:28.768357 16649 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a14419363fd4ce798c8c96eeed03366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:28 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:39788
I20250901 20:01:28.768357 16790 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a14419363fd4ce798c8c96eeed03366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:28 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:42686
I20250901 20:01:28.768846 16957 tablet_replica.cc:331] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d: stopping tablet replica
I20250901 20:01:28.769006 16957 raft_consensus.cc:2241] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:01:28.769331 16957 raft_consensus.cc:2270] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:28.769589 16959 tablet_replica.cc:331] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: stopping tablet replica
I20250901 20:01:28.769703 16959 raft_consensus.cc:2241] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:28.769814 16959 raft_consensus.cc:2270] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:28.770139 16958 tablet_replica.cc:331] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb: stopping tablet replica
I20250901 20:01:28.770318 16958 raft_consensus.cc:2241] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:28.770375 16957 ts_tablet_manager.cc:1905] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:28.770493 16958 raft_consensus.cc:2270] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:28.771282 16959 ts_tablet_manager.cc:1905] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:28.771318 16958 ts_tablet_manager.cc:1905] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:28.773300 16957 ts_tablet_manager.cc:1918] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250901 20:01:28.773437 16957 log.cc:1199] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/wal/wals/9a14419363fd4ce798c8c96eeed03366
I20250901 20:01:28.773814 16957 ts_tablet_manager.cc:1939] T 9a14419363fd4ce798c8c96eeed03366 P 6ca3083eb2e1425a828f5f42e283d36d: Deleting consensus metadata
I20250901 20:01:28.774102 16958 ts_tablet_manager.cc:1918] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250901 20:01:28.774223 16958 log.cc:1199] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/9a14419363fd4ce798c8c96eeed03366
I20250901 20:01:28.774379 16959 ts_tablet_manager.cc:1918] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250901 20:01:28.774545 16959 log.cc:1199] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/9a14419363fd4ce798c8c96eeed03366
I20250901 20:01:28.774605 16958 ts_tablet_manager.cc:1939] T 9a14419363fd4ce798c8c96eeed03366 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Deleting consensus metadata
I20250901 20:01:28.774873 16959 ts_tablet_manager.cc:1939] T 9a14419363fd4ce798c8c96eeed03366 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Deleting consensus metadata
I20250901 20:01:28.774959 16352 catalog_manager.cc:4928] TS 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477): tablet 9a14419363fd4ce798c8c96eeed03366 (table test-table [id=8bb00436df944e4493b36057392107b9]) successfully deleted
I20250901 20:01:28.775365 16355 catalog_manager.cc:4928] TS 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739): tablet 9a14419363fd4ce798c8c96eeed03366 (table test-table [id=8bb00436df944e4493b36057392107b9]) successfully deleted
I20250901 20:01:28.775501 16352 catalog_manager.cc:4928] TS 428f4b4d9c224b7bb0e9b1610fc0f8dd (127.14.67.67:39025): tablet 9a14419363fd4ce798c8c96eeed03366 (table test-table [id=8bb00436df944e4493b36057392107b9]) successfully deleted
Sep 01 20:01:28 dist-test-slave-2s4d krb5kdc[16320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756888, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 01 20:01:28 dist-test-slave-2s4d krb5kdc[16320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756888, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:28.797008 16967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:28.785031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.14.67.126:38019)
0901 20:01:28.785275 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:28.785279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:28.785293 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:01:28.785330 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:28.785333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:28.785375 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:28.785458 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:28.786162 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.786723 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:28.787307 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.787429 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:28.794288 (+ 6859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:28.794308 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:28.794310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:28.794336 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:28.795792 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:28.796127 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:28.796129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:28.796131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:28.796163 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:28.796319 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:28.796321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:28.796322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:28.796468 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:28.796557 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:28.796709 (+ 152us) server_negotiation.cc:299] Negotiation successful
0901 20:01:28.796802 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
I20250901 20:01:28.799139 16363 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 20:01:28.799525 16363 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:01:28.803807 16507 tablet_service.cc:1468] Processing CreateTablet for tablet ead1dfaa8eea4fd482b17ab8bc00e36c (DEFAULT_TABLE table=test-table [id=5b0774c6f17c42098eda35e32889faab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:28.803897 16649 tablet_service.cc:1468] Processing CreateTablet for tablet ead1dfaa8eea4fd482b17ab8bc00e36c (DEFAULT_TABLE table=test-table [id=5b0774c6f17c42098eda35e32889faab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:28.803942 16790 tablet_service.cc:1468] Processing CreateTablet for tablet ead1dfaa8eea4fd482b17ab8bc00e36c (DEFAULT_TABLE table=test-table [id=5b0774c6f17c42098eda35e32889faab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:28.804113 16507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead1dfaa8eea4fd482b17ab8bc00e36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:28.804200 16649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead1dfaa8eea4fd482b17ab8bc00e36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:28.804204 16790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead1dfaa8eea4fd482b17ab8bc00e36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:28.805922 16968 tablet_bootstrap.cc:492] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb: Bootstrap starting.
I20250901 20:01:28.806561 16970 tablet_bootstrap.cc:492] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Bootstrap starting.
I20250901 20:01:28.806907 16968 tablet_bootstrap.cc:654] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:28.807389 16970 tablet_bootstrap.cc:654] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:28.807652 16969 tablet_bootstrap.cc:492] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d: Bootstrap starting.
I20250901 20:01:28.808341 16969 tablet_bootstrap.cc:654] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:28.808341 16970 tablet_bootstrap.cc:492] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd: No bootstrap required, opened a new log
I20250901 20:01:28.808422 16970 ts_tablet_manager.cc:1397] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250901 20:01:28.808780 16970 raft_consensus.cc:357] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.808902 16970 raft_consensus.cc:383] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:28.808988 16970 raft_consensus.cc:738] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428f4b4d9c224b7bb0e9b1610fc0f8dd, State: Initialized, Role: FOLLOWER
I20250901 20:01:28.809163 16968 tablet_bootstrap.cc:492] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb: No bootstrap required, opened a new log
I20250901 20:01:28.809144 16970 consensus_queue.cc:260] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.809266 16968 ts_tablet_manager.cc:1397] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250901 20:01:28.809453 16970 ts_tablet_manager.cc:1428] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:01:28.809820 16968 raft_consensus.cc:357] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.809927 16968 raft_consensus.cc:383] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:28.809968 16968 raft_consensus.cc:738] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ed5d6178c447ce820fd6a9bd5b6ecb, State: Initialized, Role: FOLLOWER
I20250901 20:01:28.810094 16968 consensus_queue.cc:260] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.810391 16968 ts_tablet_manager.cc:1428] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250901 20:01:28.810897 16969 tablet_bootstrap.cc:492] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d: No bootstrap required, opened a new log
I20250901 20:01:28.810985 16969 ts_tablet_manager.cc:1397] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250901 20:01:28.811276 16974 raft_consensus.cc:491] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:01:28.811383 16974 raft_consensus.cc:513] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.811401 16969 raft_consensus.cc:357] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.811498 16969 raft_consensus.cc:383] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:28.811563 16969 raft_consensus.cc:738] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca3083eb2e1425a828f5f42e283d36d, State: Initialized, Role: FOLLOWER
I20250901 20:01:28.811745 16969 consensus_queue.cc:260] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.812058 16969 ts_tablet_manager.cc:1428] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:01:28.812594 16974 leader_election.cc:290] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477), 428f4b4d9c224b7bb0e9b1610fc0f8dd (127.14.67.67:39025)
I20250901 20:01:28.816047 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:28.812738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:33477 (local address 127.14.67.66:33727)
0901 20:01:28.812927 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:28.812940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:28.813010 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:28.813734 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:28.813737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:28.813757 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:28.813982 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:28.813988 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.814881 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:28.814885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:28.815685 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:28.815694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.815772 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:28.815795 (+ 23us) client_negotiation.cc:769] Sending connection context
0901 20:01:28.815838 (+ 43us) client_negotiation.cc:240] Negotiation successful
0901 20:01:28.815903 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20250901 20:01:28.816258 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:28.813020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:39025 (local address 127.14.67.66:33819)
0901 20:01:28.813181 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:28.813191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:28.813254 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:28.813573 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:28.813576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:28.813597 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:28.814226 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:28.814236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.815192 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:28.815194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:28.815900 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:28.815909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.816006 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:28.816024 (+ 18us) client_negotiation.cc:769] Sending connection context
0901 20:01:28.816067 (+ 43us) client_negotiation.cc:240] Negotiation successful
0901 20:01:28.816121 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20250901 20:01:28.816870 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:28.813304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:33727 (local address 127.14.67.65:33477)
0901 20:01:28.813543 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:28.813547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:28.813558 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:01:28.813599 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:28.813604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:28.813641 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:28.813714 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:28.814083 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.814758 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:28.816235 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.816660 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:28.816700 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:28.816745 (+ 45us) server_negotiation.cc:299] Negotiation successful
0901 20:01:28.816787 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20250901 20:01:28.816942 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:28.813093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:33819 (local address 127.14.67.67:39025)
0901 20:01:28.813339 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:28.813343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:28.813354 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:01:28.813390 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:28.813392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:28.813436 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:28.813516 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:28.814338 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.815104 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:28.816413 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.816745 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:28.816772 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:28.816808 (+ 36us) server_negotiation.cc:299] Negotiation successful
0901 20:01:28.816846 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
I20250901 20:01:28.817203 16527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ead1dfaa8eea4fd482b17ab8bc00e36c" candidate_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca3083eb2e1425a828f5f42e283d36d" is_pre_election: true
I20250901 20:01:28.817324 16527 raft_consensus.cc:2466] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ed5d6178c447ce820fd6a9bd5b6ecb in term 0.
I20250901 20:01:28.817306 16811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ead1dfaa8eea4fd482b17ab8bc00e36c" candidate_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" is_pre_election: true
I20250901 20:01:28.817418 16811 raft_consensus.cc:2466] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ed5d6178c447ce820fd6a9bd5b6ecb in term 0.
I20250901 20:01:28.817660 16600 leader_election.cc:304] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ed5d6178c447ce820fd6a9bd5b6ecb, 6ca3083eb2e1425a828f5f42e283d36d; no voters:
I20250901 20:01:28.817862 16974 raft_consensus.cc:2802] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:01:28.817912 16974 raft_consensus.cc:491] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:01:28.817941 16974 raft_consensus.cc:3058] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:28.818727 16974 raft_consensus.cc:513] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.818998 16974 leader_election.cc:290] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [CANDIDATE]: Term 1 election: Requested vote from peers 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477), 428f4b4d9c224b7bb0e9b1610fc0f8dd (127.14.67.67:39025)
I20250901 20:01:28.819347 16811 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ead1dfaa8eea4fd482b17ab8bc00e36c" candidate_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd"
I20250901 20:01:28.819396 16527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ead1dfaa8eea4fd482b17ab8bc00e36c" candidate_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca3083eb2e1425a828f5f42e283d36d"
I20250901 20:01:28.819451 16811 raft_consensus.cc:3058] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:28.819478 16527 raft_consensus.cc:3058] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:28.820397 16527 raft_consensus.cc:2466] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ed5d6178c447ce820fd6a9bd5b6ecb in term 1.
I20250901 20:01:28.820390 16811 raft_consensus.cc:2466] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ed5d6178c447ce820fd6a9bd5b6ecb in term 1.
I20250901 20:01:28.820688 16600 leader_election.cc:304] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ed5d6178c447ce820fd6a9bd5b6ecb, 428f4b4d9c224b7bb0e9b1610fc0f8dd; no voters:
I20250901 20:01:28.820914 16974 raft_consensus.cc:2802] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:28.821081 16974 raft_consensus.cc:695] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 LEADER]: Becoming Leader. State: Replica: 14ed5d6178c447ce820fd6a9bd5b6ecb, State: Running, Role: LEADER
I20250901 20:01:28.821296 16974 consensus_queue.cc:237] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.823012 16363 catalog_manager.cc:5582] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb reported cstate change: term changed from 0 to 1, leader changed from <none> to 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66). New cstate: current_term: 1 leader_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20250901 20:01:28.840102 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:28.837244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.14.67.66:46739)
0901 20:01:28.837493 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:28.837496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:28.837508 (+ 12us) server_negotiation.cc:407] Connection header received
0901 20:01:28.837574 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:28.837576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:28.837625 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:28.837723 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:28.837991 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.838483 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:28.839089 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.839223 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:28.839269 (+ 46us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:28.839725 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:28.839811 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:28.839891 (+ 80us) server_negotiation.cc:299] Negotiation successful
0901 20:01:28.839932 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":114,"threads_started":1}
I20250901 20:01:28.848424 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:28.846011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:51969 (local address 127.14.67.67:39025)
0901 20:01:28.846141 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:28.846145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:28.846158 (+ 13us) server_negotiation.cc:407] Connection header received
0901 20:01:28.846252 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:28.846256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:28.846296 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:28.846359 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:28.846744 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.847183 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:28.847813 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.847994 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:28.848035 (+ 41us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:28.848094 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:28.848158 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:28.848258 (+ 100us) server_negotiation.cc:299] Negotiation successful
0901 20:01:28.848299 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20250901 20:01:28.848438 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:28.845809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:39025 (local address 127.14.67.66:51969)
0901 20:01:28.846059 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:28.846075 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:28.846164 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:28.846378 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:28.846381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:28.846388 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0901 20:01:28.846649 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:28.846655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.847275 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:28.847279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:28.847714 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:28.847721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.847818 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:28.847859 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0901 20:01:28.848174 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0901 20:01:28.848179 (+ 5us) client_negotiation.cc:769] Sending connection context
0901 20:01:28.848232 (+ 53us) client_negotiation.cc:240] Negotiation successful
0901 20:01:28.848300 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
I20250901 20:01:28.853865 16811 raft_consensus.cc:1273] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Refusing update from remote peer 14ed5d6178c447ce820fd6a9bd5b6ecb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:28.853892 16527 raft_consensus.cc:1273] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Refusing update from remote peer 14ed5d6178c447ce820fd6a9bd5b6ecb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:28.854251 16979 consensus_queue.cc:1035] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [LEADER]: Connected to new peer: Peer: permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:28.854406 16974 consensus_queue.cc:1035] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:28.869092 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:28.866343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:46739 (local address 127.14.67.67:41373)
0901 20:01:28.866598 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:28.866609 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:28.866698 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:28.866971 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:28.866975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:28.866984 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0901 20:01:28.867174 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:28.867179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.867879 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:28.867882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:28.868313 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:28.868321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.868420 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:28.868450 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0901 20:01:28.868849 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0901 20:01:28.868854 (+ 5us) client_negotiation.cc:769] Sending connection context
0901 20:01:28.868926 (+ 72us) client_negotiation.cc:240] Negotiation successful
0901 20:01:28.868986 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20250901 20:01:28.869117 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:28.866397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:41373 (local address 127.14.67.66:46739)
0901 20:01:28.866523 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:28.866528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:28.866666 (+ 138us) server_negotiation.cc:407] Connection header received
0901 20:01:28.866782 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:28.866785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:28.866829 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:28.866892 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:28.867287 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.867776 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:28.868428 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:28.868618 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:28.868683 (+ 65us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:28.868761 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:28.868842 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:28.868946 (+ 104us) server_negotiation.cc:299] Negotiation successful
0901 20:01:28.868997 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20250901 20:01:28.889847 16367 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35488:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:28.889982 16367 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35488:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:28.891881 16367 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de: Sending DeleteTablet for 3 replicas of tablet ead1dfaa8eea4fd482b17ab8bc00e36c
I20250901 20:01:28.892396 16507 tablet_service.cc:1515] Processing DeleteTablet for tablet ead1dfaa8eea4fd482b17ab8bc00e36c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:28 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:53598
I20250901 20:01:28.892449 16649 tablet_service.cc:1515] Processing DeleteTablet for tablet ead1dfaa8eea4fd482b17ab8bc00e36c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:28 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:39788
I20250901 20:01:28.892580 16958 tablet_replica.cc:331] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb: stopping tablet replica
I20250901 20:01:28.892580 16957 tablet_replica.cc:331] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d: stopping tablet replica
I20250901 20:01:28.892683 16958 raft_consensus.cc:2241] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:01:28.892690 16957 raft_consensus.cc:2241] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:28.892786 16957 raft_consensus.cc:2270] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:28.892940 16958 raft_consensus.cc:2270] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:28.893126 16367 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 20:01:28.893286 16790 tablet_service.cc:1515] Processing DeleteTablet for tablet ead1dfaa8eea4fd482b17ab8bc00e36c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:28 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:42686
I20250901 20:01:28.893595 16959 tablet_replica.cc:331] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd: stopping tablet replica
I20250901 20:01:28.893707 16959 raft_consensus.cc:2241] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:28.893803 16959 raft_consensus.cc:2270] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:28.893810 16958 ts_tablet_manager.cc:1905] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:28.893842 16957 ts_tablet_manager.cc:1905] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:28.894570 16959 ts_tablet_manager.cc:1905] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:28.896129 16957 ts_tablet_manager.cc:1918] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:01:28.896129 16958 ts_tablet_manager.cc:1918] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:01:28.896193 16958 log.cc:1199] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/ead1dfaa8eea4fd482b17ab8bc00e36c
I20250901 20:01:28.896193 16957 log.cc:1199] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-0/wal/wals/ead1dfaa8eea4fd482b17ab8bc00e36c
I20250901 20:01:28.896461 16958 ts_tablet_manager.cc:1939] T ead1dfaa8eea4fd482b17ab8bc00e36c P 14ed5d6178c447ce820fd6a9bd5b6ecb: Deleting consensus metadata
I20250901 20:01:28.896461 16957 ts_tablet_manager.cc:1939] T ead1dfaa8eea4fd482b17ab8bc00e36c P 6ca3083eb2e1425a828f5f42e283d36d: Deleting consensus metadata
I20250901 20:01:28.896955 16959 ts_tablet_manager.cc:1918] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:01:28.897015 16959 log.cc:1199] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/ead1dfaa8eea4fd482b17ab8bc00e36c
I20250901 20:01:28.897250 16959 ts_tablet_manager.cc:1939] T ead1dfaa8eea4fd482b17ab8bc00e36c P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Deleting consensus metadata
I20250901 20:01:28.897120 16355 catalog_manager.cc:4928] TS 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739): tablet ead1dfaa8eea4fd482b17ab8bc00e36c (table test-table [id=5b0774c6f17c42098eda35e32889faab]) successfully deleted
I20250901 20:01:28.897120 16352 catalog_manager.cc:4928] TS 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477): tablet ead1dfaa8eea4fd482b17ab8bc00e36c (table test-table [id=5b0774c6f17c42098eda35e32889faab]) successfully deleted
I20250901 20:01:28.898017 16352 catalog_manager.cc:4928] TS 428f4b4d9c224b7bb0e9b1610fc0f8dd (127.14.67.67:39025): tablet ead1dfaa8eea4fd482b17ab8bc00e36c (table test-table [id=5b0774c6f17c42098eda35e32889faab]) successfully deleted
W20250901 20:01:28.898137 16367 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:01:28.902132 16507 tablet_service.cc:1468] Processing CreateTablet for tablet 2cef69a3cb25414ea2badc1a2b0b9ab8 (DEFAULT_TABLE table=test-table [id=1bdbb8fd9909461ea429a6a921ef6b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:28.902374 16507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cef69a3cb25414ea2badc1a2b0b9ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:28.902457 16790 tablet_service.cc:1468] Processing CreateTablet for tablet 2cef69a3cb25414ea2badc1a2b0b9ab8 (DEFAULT_TABLE table=test-table [id=1bdbb8fd9909461ea429a6a921ef6b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:28.902691 16790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cef69a3cb25414ea2badc1a2b0b9ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:28.904721 16969 tablet_bootstrap.cc:492] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d: Bootstrap starting.
I20250901 20:01:28.905385 16969 tablet_bootstrap.cc:654] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:28.905900 16970 tablet_bootstrap.cc:492] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Bootstrap starting.
I20250901 20:01:28.906589 16970 tablet_bootstrap.cc:654] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:28.907600 16970 tablet_bootstrap.cc:492] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: No bootstrap required, opened a new log
I20250901 20:01:28.907677 16970 ts_tablet_manager.cc:1397] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250901 20:01:28.908015 16970 raft_consensus.cc:357] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.908099 16970 raft_consensus.cc:383] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:28.908133 16970 raft_consensus.cc:738] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428f4b4d9c224b7bb0e9b1610fc0f8dd, State: Initialized, Role: FOLLOWER
I20250901 20:01:28.908248 16970 consensus_queue.cc:260] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.908524 16970 ts_tablet_manager.cc:1428] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250901 20:01:28.909206 16969 tablet_bootstrap.cc:492] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d: No bootstrap required, opened a new log
I20250901 20:01:28.909284 16969 ts_tablet_manager.cc:1397] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250901 20:01:28.909746 16969 raft_consensus.cc:357] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.909854 16969 raft_consensus.cc:383] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:28.909890 16969 raft_consensus.cc:738] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca3083eb2e1425a828f5f42e283d36d, State: Initialized, Role: FOLLOWER
I20250901 20:01:28.910009 16969 consensus_queue.cc:260] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.910247 16969 ts_tablet_manager.cc:1428] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250901 20:01:28.910799 16649 tablet_service.cc:1468] Processing CreateTablet for tablet 2cef69a3cb25414ea2badc1a2b0b9ab8 (DEFAULT_TABLE table=test-table [id=1bdbb8fd9909461ea429a6a921ef6b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:28.911074 16649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cef69a3cb25414ea2badc1a2b0b9ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:28.912743 16968 tablet_bootstrap.cc:492] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Bootstrap starting.
I20250901 20:01:28.913841 16968 tablet_bootstrap.cc:654] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:28.915777 16968 tablet_bootstrap.cc:492] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb: No bootstrap required, opened a new log
I20250901 20:01:28.915856 16968 ts_tablet_manager.cc:1397] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250901 20:01:28.916222 16968 raft_consensus.cc:357] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.916307 16968 raft_consensus.cc:383] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:28.916342 16968 raft_consensus.cc:738] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ed5d6178c447ce820fd6a9bd5b6ecb, State: Initialized, Role: FOLLOWER
I20250901 20:01:28.916460 16968 consensus_queue.cc:260] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:28.916692 16968 ts_tablet_manager.cc:1428] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250901 20:01:29.041745 16981 raft_consensus.cc:491] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:01:29.041886 16981 raft_consensus.cc:513] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:29.042332 16981 leader_election.cc:290] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477), 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739)
I20250901 20:01:29.042790 16669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cef69a3cb25414ea2badc1a2b0b9ab8" candidate_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" is_pre_election: true
I20250901 20:01:29.042797 16515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cef69a3cb25414ea2badc1a2b0b9ab8" candidate_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca3083eb2e1425a828f5f42e283d36d" is_pre_election: true
I20250901 20:01:29.042935 16669 raft_consensus.cc:2466] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428f4b4d9c224b7bb0e9b1610fc0f8dd in term 0.
I20250901 20:01:29.042958 16515 raft_consensus.cc:2466] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428f4b4d9c224b7bb0e9b1610fc0f8dd in term 0.
I20250901 20:01:29.043251 16742 leader_election.cc:304] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428f4b4d9c224b7bb0e9b1610fc0f8dd, 6ca3083eb2e1425a828f5f42e283d36d; no voters:
I20250901 20:01:29.043474 16981 raft_consensus.cc:2802] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:01:29.043519 16981 raft_consensus.cc:491] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:01:29.043565 16981 raft_consensus.cc:3058] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:29.044440 16981 raft_consensus.cc:513] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:29.044756 16981 leader_election.cc:290] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [CANDIDATE]: Term 1 election: Requested vote from peers 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477), 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739)
I20250901 20:01:29.045082 16515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cef69a3cb25414ea2badc1a2b0b9ab8" candidate_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca3083eb2e1425a828f5f42e283d36d"
I20250901 20:01:29.045176 16515 raft_consensus.cc:3058] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:29.045976 16515 raft_consensus.cc:2466] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428f4b4d9c224b7bb0e9b1610fc0f8dd in term 1.
I20250901 20:01:29.046186 16669 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cef69a3cb25414ea2badc1a2b0b9ab8" candidate_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb"
I20250901 20:01:29.046240 16742 leader_election.cc:304] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428f4b4d9c224b7bb0e9b1610fc0f8dd, 6ca3083eb2e1425a828f5f42e283d36d; no voters:
I20250901 20:01:29.046300 16669 raft_consensus.cc:3058] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:29.046383 16981 raft_consensus.cc:2802] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:29.046490 16981 raft_consensus.cc:695] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 LEADER]: Becoming Leader. State: Replica: 428f4b4d9c224b7bb0e9b1610fc0f8dd, State: Running, Role: LEADER
I20250901 20:01:29.046648 16981 consensus_queue.cc:237] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } }
I20250901 20:01:29.047235 16669 raft_consensus.cc:2466] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428f4b4d9c224b7bb0e9b1610fc0f8dd in term 1.
I20250901 20:01:29.048218 16367 catalog_manager.cc:5582] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 428f4b4d9c224b7bb0e9b1610fc0f8dd (127.14.67.67). New cstate: current_term: 1 leader_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428f4b4d9c224b7bb0e9b1610fc0f8dd" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 39025 } health_report { overall_health: HEALTHY } } }
I20250901 20:01:29.067839 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:29.065391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.14.67.67:39025)
0901 20:01:29.065549 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:29.065553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:29.065566 (+ 13us) server_negotiation.cc:407] Connection header received
0901 20:01:29.065682 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:29.065685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:29.065729 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:29.065787 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:29.066117 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:29.066610 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:29.067201 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:29.067370 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:29.067410 (+ 40us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:29.067491 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:29.067553 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:29.067671 (+ 118us) server_negotiation.cc:299] Negotiation successful
0901 20:01:29.067713 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250901 20:01:29.073047 16669 raft_consensus.cc:1273] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Refusing update from remote peer 428f4b4d9c224b7bb0e9b1610fc0f8dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:29.073338 16515 raft_consensus.cc:1273] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 6ca3083eb2e1425a828f5f42e283d36d [term 1 FOLLOWER]: Refusing update from remote peer 428f4b4d9c224b7bb0e9b1610fc0f8dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:29.073398 16918 consensus_queue.cc:1035] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ed5d6178c447ce820fd6a9bd5b6ecb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 46739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:29.073676 16981 consensus_queue.cc:1035] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca3083eb2e1425a828f5f42e283d36d" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 20:01:29.085382 16937 tablet_replica.cc:1304] Aborted: operation has been aborted: cancelling pending write operations
I20250901 20:01:30.629477 16363 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35488:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:30.629643 16363 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35488:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:30.631734 16363 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1b4080a8180f45afa29d779ceaacc1de: Sending DeleteTablet for 3 replicas of tablet 2cef69a3cb25414ea2badc1a2b0b9ab8
I20250901 20:01:30.632386 16649 tablet_service.cc:1515] Processing DeleteTablet for tablet 2cef69a3cb25414ea2badc1a2b0b9ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:30 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:39788
I20250901 20:01:30.632644 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 16442
I20250901 20:01:30.632683 16507 tablet_service.cc:1515] Processing DeleteTablet for tablet 2cef69a3cb25414ea2badc1a2b0b9ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:30 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:53598
I20250901 20:01:30.633164 16790 tablet_service.cc:1515] Processing DeleteTablet for tablet 2cef69a3cb25414ea2badc1a2b0b9ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:30 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:42686
I20250901 20:01:30.633724 17020 tablet_replica.cc:331] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb: stopping tablet replica
I20250901 20:01:30.633998 17021 tablet_replica.cc:331] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: stopping tablet replica
I20250901 20:01:30.634156 17021 raft_consensus.cc:2241] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:01:30.634164 17020 raft_consensus.cc:2241] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:30.634325 17020 raft_consensus.cc:2270] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:30.634403 17021 raft_consensus.cc:2270] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:30.635013 17021 ts_tablet_manager.cc:1905] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:30.635062 17020 ts_tablet_manager.cc:1905] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:30.637406 17021 ts_tablet_manager.cc:1918] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250901 20:01:30.637540 17021 log.cc:1199] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/2cef69a3cb25414ea2badc1a2b0b9ab8
I20250901 20:01:30.637893 17021 ts_tablet_manager.cc:1939] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 428f4b4d9c224b7bb0e9b1610fc0f8dd: Deleting consensus metadata
I20250901 20:01:30.638163 17020 ts_tablet_manager.cc:1918] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20250901 20:01:30.638278 17020 log.cc:1199] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/2cef69a3cb25414ea2badc1a2b0b9ab8
I20250901 20:01:30.638571 16352 catalog_manager.cc:4928] TS 428f4b4d9c224b7bb0e9b1610fc0f8dd (127.14.67.67:39025): tablet 2cef69a3cb25414ea2badc1a2b0b9ab8 (table test-table [id=1bdbb8fd9909461ea429a6a921ef6b7f]) successfully deleted
I20250901 20:01:30.638665 17020 ts_tablet_manager.cc:1939] T 2cef69a3cb25414ea2badc1a2b0b9ab8 P 14ed5d6178c447ce820fd6a9bd5b6ecb: Deleting consensus metadata
I20250901 20:01:30.639309 16355 catalog_manager.cc:4928] TS 14ed5d6178c447ce820fd6a9bd5b6ecb (127.14.67.66:46739): tablet 2cef69a3cb25414ea2badc1a2b0b9ab8 (table test-table [id=1bdbb8fd9909461ea429a6a921ef6b7f]) successfully deleted
W20250901 20:01:30.642072 16352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.67.65:33477 (error 108)
I20250901 20:01:30.642146 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 16584
I20250901 20:01:30.644579 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:30.644211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:33477 (local address 127.0.0.1:53612)
0901 20:01:30.644453 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:30.644504 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.65:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
W20250901 20:01:30.644699 16352 catalog_manager.cc:4671] TS 6ca3083eb2e1425a828f5f42e283d36d (127.14.67.65:33477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cef69a3cb25414ea2badc1a2b0b9ab8: Network error: Client connection negotiation failed: client connection to 127.14.67.65:33477: connect: Connection refused (error 111)
I20250901 20:01:30.649111 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 16726
I20250901 20:01:30.655428 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 16336
2025-09-01T20:01:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7009 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:01:30 dist-test-slave-2s4d krb5kdc[17026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:01:30 dist-test-slave-2s4d krb5kdc[17026](info): set up 2 sockets
Sep 01 20:01:30 dist-test-slave-2s4d krb5kdc[17026](info): commencing operation
krb5kdc: starting...
W20250901 20:01:32.702878 14605 mini_kdc.cc:121] Time spent starting KDC: real 2.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:01:32 dist-test-slave-2s4d krb5kdc[17026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:01:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:01:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:32.829207 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:35409
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:35409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:32.928676 17042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:32.928871 17042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:32.928922 17042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:32.931900 17042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:01:32.931947 17042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:32.931969 17042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:32.931988 17042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:01:32.932008 17042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:01:32.936071 17042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:35409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:35409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:32.936939 17042 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:32.937669 17042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:32.943073 17050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:32.943154 17042 server_base.cc:1047] running on GCE node
W20250901 20:01:32.943070 17047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:32.943068 17048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:32.943801 17042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:32.944558 17042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:32.945724 17042 hybrid_clock.cc:648] HybridClock initialized: now 1756756892945708 us; error 32 us; skew 500 ppm
Sep 01 20:01:32 dist-test-slave-2s4d krb5kdc[17026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756892, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:32.948238 17042 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:01:32.949191 17042 webserver.cc:480] Webserver started at http://127.14.67.126:40433/ using document root <none> and password file <none>
I20250901 20:01:32.949692 17042 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:32.949745 17042 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:32.949913 17042 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:32.951524 17042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "fd029b0f7a8a469d9c0033088f4c1715"
format_stamp: "Formatted at 2025-09-01 20:01:32 on dist-test-slave-2s4d"
server_key: "cdac8a0a99b140cef406c198e16015d2"
server_key_iv: "d7b8fb9323493d526fac7aac8a331a1e"
server_key_version: "encryptionkey@0"
I20250901 20:01:32.951975 17042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "fd029b0f7a8a469d9c0033088f4c1715"
format_stamp: "Formatted at 2025-09-01 20:01:32 on dist-test-slave-2s4d"
server_key: "cdac8a0a99b140cef406c198e16015d2"
server_key_iv: "d7b8fb9323493d526fac7aac8a331a1e"
server_key_version: "encryptionkey@0"
I20250901 20:01:32.954995 17042 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:01:32.956912 17057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:32.957851 17042 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:32.957952 17042 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "fd029b0f7a8a469d9c0033088f4c1715"
format_stamp: "Formatted at 2025-09-01 20:01:32 on dist-test-slave-2s4d"
server_key: "cdac8a0a99b140cef406c198e16015d2"
server_key_iv: "d7b8fb9323493d526fac7aac8a331a1e"
server_key_version: "encryptionkey@0"
I20250901 20:01:32.958029 17042 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:32.980795 17042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:32.983522 17042 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:32.983687 17042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:32.991055 17042 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:35409
I20250901 20:01:32.991060 17109 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:35409 every 8 connection(s)
I20250901 20:01:32.992002 17042 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:01:32.993942 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 17042
I20250901 20:01:32.994061 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:01:32.994316 14605 external_mini_cluster.cc:1468] Setting key e786a020b39b6ae4de2cebb2cb4a3ff8
I20250901 20:01:32.994522 17110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:33.000227 17110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715: Bootstrap starting.
Sep 01 20:01:32 dist-test-slave-2s4d krb5kdc[17026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:33.002645 17110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:33.003371 17110 log.cc:826] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:33.005244 17110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715: No bootstrap required, opened a new log
I20250901 20:01:33.006412 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:32.995503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.14.67.126:35409)
0901 20:01:32.995884 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:32.995892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:32.995923 (+ 31us) server_negotiation.cc:407] Connection header received
0901 20:01:32.996542 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:32.996558 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:32.996830 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:32.997104 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:32.997990 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:32.998754 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:32.999350 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:32.999612 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:33.002033 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:33.002062 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:33.002072 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:33.002101 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:33.004109 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:33.004537 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:33.004541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:33.004544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:33.004604 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:33.004858 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:33.004861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:33.004862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:33.005154 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:33.005333 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:33.005614 (+ 281us) server_negotiation.cc:299] Negotiation successful
0901 20:01:33.005867 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":125,"threads_started":1}
I20250901 20:01:33.007714 17110 raft_consensus.cc:357] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd029b0f7a8a469d9c0033088f4c1715" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35409 } }
I20250901 20:01:33.007885 17110 raft_consensus.cc:383] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:33.007949 17110 raft_consensus.cc:738] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd029b0f7a8a469d9c0033088f4c1715, State: Initialized, Role: FOLLOWER
I20250901 20:01:33.008394 17110 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd029b0f7a8a469d9c0033088f4c1715" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35409 } }
I20250901 20:01:33.008517 17110 raft_consensus.cc:397] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:01:33.008561 17110 raft_consensus.cc:491] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:01:33.008638 17110 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:33.009373 17110 raft_consensus.cc:513] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd029b0f7a8a469d9c0033088f4c1715" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35409 } }
I20250901 20:01:33.009660 17110 leader_election.cc:304] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd029b0f7a8a469d9c0033088f4c1715; no voters:
I20250901 20:01:33.009874 17110 leader_election.cc:290] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:01:33.010190 17115 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:33.010396 17115 raft_consensus.cc:695] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [term 1 LEADER]: Becoming Leader. State: Replica: fd029b0f7a8a469d9c0033088f4c1715, State: Running, Role: LEADER
I20250901 20:01:33.010703 17115 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd029b0f7a8a469d9c0033088f4c1715" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35409 } }
I20250901 20:01:33.010883 17110 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:01:33.012220 17117 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd029b0f7a8a469d9c0033088f4c1715. Latest consensus state: current_term: 1 leader_uuid: "fd029b0f7a8a469d9c0033088f4c1715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd029b0f7a8a469d9c0033088f4c1715" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35409 } } }
I20250901 20:01:33.012319 17117 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:33.012316 17116 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd029b0f7a8a469d9c0033088f4c1715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd029b0f7a8a469d9c0033088f4c1715" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35409 } } }
I20250901 20:01:33.012627 17116 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:33.012689 17124 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:01:33.015400 17124 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:01:33.020977 17124 catalog_manager.cc:1349] Generated new cluster ID: 414c176783b14ae188c3779ab943980f
I20250901 20:01:33.021106 17124 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:01:33.034264 17124 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:01:33.035122 17124 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:01:33.042474 17124 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fd029b0f7a8a469d9c0033088f4c1715: Generated new TSK 0
I20250901 20:01:33.043097 17124 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:33.093837 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:35409
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:33.192492 17138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:33.192662 17138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:33.192714 17138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:33.195676 17138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:33.195724 17138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:33.195796 17138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:01:33.199615 17138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:33.200417 17138 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:33.201134 17138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:33.206842 17144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:33.206988 17143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:33.207255 17138 server_base.cc:1047] running on GCE node
W20250901 20:01:33.206928 17146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:33.207788 17138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:33.208292 17138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:33.209479 17138 hybrid_clock.cc:648] HybridClock initialized: now 1756756893209470 us; error 45 us; skew 500 ppm
Sep 01 20:01:33 dist-test-slave-2s4d krb5kdc[17026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756893, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:33.212055 17138 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:01:33.212966 17138 webserver.cc:480] Webserver started at http://127.14.67.65:34541/ using document root <none> and password file <none>
I20250901 20:01:33.213456 17138 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:33.213502 17138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:33.213685 17138 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:33.215282 17138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "97d216650a1d42ab80a87eecb6511694"
format_stamp: "Formatted at 2025-09-01 20:01:33 on dist-test-slave-2s4d"
server_key: "3df70bc0b60b654e8c9a82f00699506b"
server_key_iv: "39a2d476d64f1e1a4adf92624bd4bebd"
server_key_version: "encryptionkey@0"
I20250901 20:01:33.215709 17138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "97d216650a1d42ab80a87eecb6511694"
format_stamp: "Formatted at 2025-09-01 20:01:33 on dist-test-slave-2s4d"
server_key: "3df70bc0b60b654e8c9a82f00699506b"
server_key_iv: "39a2d476d64f1e1a4adf92624bd4bebd"
server_key_version: "encryptionkey@0"
I20250901 20:01:33.218708 17138 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:01:33.220575 17153 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:33.221550 17138 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:33.221647 17138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "97d216650a1d42ab80a87eecb6511694"
format_stamp: "Formatted at 2025-09-01 20:01:33 on dist-test-slave-2s4d"
server_key: "3df70bc0b60b654e8c9a82f00699506b"
server_key_iv: "39a2d476d64f1e1a4adf92624bd4bebd"
server_key_version: "encryptionkey@0"
I20250901 20:01:33.221726 17138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:33.247049 17138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:33.249545 17138 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:33.249706 17138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:33.250187 17138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:33.251020 17138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:33.251066 17138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:33.251121 17138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:33.251147 17138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:33.259205 17138 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:43479
I20250901 20:01:33.259238 17266 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:43479 every 8 connection(s)
I20250901 20:01:33.260046 17138 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
Sep 01 20:01:33 dist-test-slave-2s4d krb5kdc[17026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756893, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:33.268615 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 17138
I20250901 20:01:33.268733 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:01:33.269006 14605 external_mini_cluster.cc:1468] Setting key 17dd21ea9c214f64a6b0a8da2cb37a41
I20250901 20:01:33.271885 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:33.261580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:43475 (local address 127.14.67.126:35409)
0901 20:01:33.261701 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:33.261705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:33.262346 (+ 641us) server_negotiation.cc:407] Connection header received
0901 20:01:33.263174 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:33.263177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:33.263221 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:33.263298 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:33.264454 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:33.264953 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:33.266275 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:33.266412 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:33.268384 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:33.268418 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:33.268423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:33.268453 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:33.269951 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:33.270482 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:33.270486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:33.270488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:33.270561 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:33.270894 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:33.270898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:33.270900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:33.271057 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:33.271159 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:33.271612 (+ 453us) server_negotiation.cc:299] Negotiation successful
0901 20:01:33.271727 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250901 20:01:33.272575 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:33.261818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:35409 (local address 127.14.67.65:43475)
0901 20:01:33.262234 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:33.262268 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:33.263007 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:33.263428 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:33.263435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:33.263806 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:33.264307 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:33.264317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:33.265076 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:33.265079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:33.266165 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:33.266173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:33.266322 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:33.266894 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:33.266916 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:33.268233 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:33.270059 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:33.270064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:33.270076 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:33.270390 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:33.270661 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:33.270663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:33.270665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:33.270769 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:33.271178 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:33.271185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:33.271389 (+ 204us) client_negotiation.cc:769] Sending connection context
0901 20:01:33.271566 (+ 177us) client_negotiation.cc:240] Negotiation successful
0901 20:01:33.271824 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
I20250901 20:01:33.273595 17267 heartbeater.cc:344] Connected to a master server at 127.14.67.126:35409
I20250901 20:01:33.273805 17267 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:33.274268 17267 heartbeater.cc:507] Master 127.14.67.126:35409 requested a full tablet report, sending...
I20250901 20:01:33.275624 17074 ts_manager.cc:194] Registered new tserver with Master: 97d216650a1d42ab80a87eecb6511694 (127.14.67.65:43479)
I20250901 20:01:33.276700 17074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:43475
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:33.313051 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:35409
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:33.412752 17274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:33.412941 17274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:33.413084 17274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:33.416169 17274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:33.416226 17274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:33.416327 17274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:01:33.420257 17274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:33.421168 17274 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:33.421924 17274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:33.427956 17279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:33.427982 17280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:33.427942 17282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:33.428158 17274 server_base.cc:1047] running on GCE node
I20250901 20:01:33.428530 17274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:33.428993 17274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:33.430135 17274 hybrid_clock.cc:648] HybridClock initialized: now 1756756893430121 us; error 35 us; skew 500 ppm
Sep 01 20:01:33 dist-test-slave-2s4d krb5kdc[17026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756893, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:33.432648 17274 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:01:33.433545 17274 webserver.cc:480] Webserver started at http://127.14.67.66:37853/ using document root <none> and password file <none>
I20250901 20:01:33.434007 17274 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:33.434053 17274 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:33.434202 17274 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:33.435761 17274 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "3689ed7d9cc24ddb8c4a8e989ce5c15c"
format_stamp: "Formatted at 2025-09-01 20:01:33 on dist-test-slave-2s4d"
server_key: "d4922306cd598fe2e6192978a8d31b65"
server_key_iv: "fbbd6b2e48bffeb4e5566b67bdcb16bc"
server_key_version: "encryptionkey@0"
I20250901 20:01:33.436157 17274 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "3689ed7d9cc24ddb8c4a8e989ce5c15c"
format_stamp: "Formatted at 2025-09-01 20:01:33 on dist-test-slave-2s4d"
server_key: "d4922306cd598fe2e6192978a8d31b65"
server_key_iv: "fbbd6b2e48bffeb4e5566b67bdcb16bc"
server_key_version: "encryptionkey@0"
I20250901 20:01:33.439146 17274 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20250901 20:01:33.440999 17289 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:33.441898 17274 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250901 20:01:33.441994 17274 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "3689ed7d9cc24ddb8c4a8e989ce5c15c"
format_stamp: "Formatted at 2025-09-01 20:01:33 on dist-test-slave-2s4d"
server_key: "d4922306cd598fe2e6192978a8d31b65"
server_key_iv: "fbbd6b2e48bffeb4e5566b67bdcb16bc"
server_key_version: "encryptionkey@0"
I20250901 20:01:33.442070 17274 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:33.452793 17274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:33.455364 17274 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:33.455530 17274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:33.456001 17274 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:33.456772 17274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:33.456820 17274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:33.456873 17274 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:33.456899 17274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:33.465289 17274 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:38323
I20250901 20:01:33.465317 17402 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:38323 every 8 connection(s)
I20250901 20:01:33.466115 17274 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:01:33.468029 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 17274
I20250901 20:01:33.468130 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:01:33.468392 14605 external_mini_cluster.cc:1468] Setting key feb8092ce773a5c8cc33035282f9314f
Sep 01 20:01:33 dist-test-slave-2s4d krb5kdc[17026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756893, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.67.67@KRBTEST.COM; defaulting to no policy
I20250901 20:01:33.478461 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:33.467704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:34925 (local address 127.14.67.126:35409)
0901 20:01:33.467878 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:33.467883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:33.468540 (+ 657us) server_negotiation.cc:407] Connection header received
0901 20:01:33.469406 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:33.469410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:33.469460 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:33.469547 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:33.471157 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:33.471952 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:33.473021 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:33.473157 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:33.475250 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:33.475263 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:33.475265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:33.475289 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:33.476598 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:33.477144 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:33.477146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:33.477148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:33.477184 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:33.477456 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:33.477458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:33.477459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:33.477578 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:33.477646 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:33.478217 (+ 571us) server_negotiation.cc:299] Negotiation successful
0901 20:01:33.478325 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250901 20:01:33.479072 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:33.467937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:35409 (local address 127.14.67.66:34925)
0901 20:01:33.468381 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:33.468425 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:33.469233 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:33.469694 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:33.469704 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:33.470202 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:33.470944 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:33.470964 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:33.472066 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:33.472069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:33.472921 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:33.472928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:33.473065 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:33.473577 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:33.473599 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:33.475102 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:33.476721 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:33.476729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:33.476740 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:33.477054 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:33.477257 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:33.477260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:33.477262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:33.477374 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:33.477669 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:33.477677 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:33.477960 (+ 283us) client_negotiation.cc:769] Sending connection context
0901 20:01:33.478171 (+ 211us) client_negotiation.cc:240] Negotiation successful
0901 20:01:33.478440 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":97,"threads_started":1}
I20250901 20:01:33.479995 17403 heartbeater.cc:344] Connected to a master server at 127.14.67.126:35409
I20250901 20:01:33.480204 17403 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:33.480670 17403 heartbeater.cc:507] Master 127.14.67.126:35409 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.67@KRBTEST.COM" created.
I20250901 20:01:33.481575 17074 ts_manager.cc:194] Registered new tserver with Master: 3689ed7d9cc24ddb8c4a8e989ce5c15c (127.14.67.66:38323)
I20250901 20:01:33.482152 17074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:34925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:33.513409 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:35409
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:33.612442 17410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:33.612617 17410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:33.612668 17410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:33.615711 17410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:33.615761 17410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:33.615833 17410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:01:33.619721 17410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:33.620575 17410 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:33.621289 17410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:33.627127 17415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:33.627131 17418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:33.627247 17410 server_base.cc:1047] running on GCE node
W20250901 20:01:33.627188 17416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:33.627656 17410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:33.628111 17410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:33.629258 17410 hybrid_clock.cc:648] HybridClock initialized: now 1756756893629242 us; error 29 us; skew 500 ppm
Sep 01 20:01:33 dist-test-slave-2s4d krb5kdc[17026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756893, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:33.631819 17410 init.cc:377] Logged in from keytab as kudu/127.14.67.67@KRBTEST.COM (short username kudu)
I20250901 20:01:33.632776 17410 webserver.cc:480] Webserver started at http://127.14.67.67:38861/ using document root <none> and password file <none>
I20250901 20:01:33.633260 17410 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:33.633307 17410 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:33.633478 17410 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:33.635106 17410 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "aca98ab52e964dd1bdb2baeb7b56abfa"
format_stamp: "Formatted at 2025-09-01 20:01:33 on dist-test-slave-2s4d"
server_key: "40ea645ca4dc10a964091ebe3a3006c4"
server_key_iv: "b4e6f015e473716b2268488fc8a7d753"
server_key_version: "encryptionkey@0"
I20250901 20:01:33.635545 17410 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "aca98ab52e964dd1bdb2baeb7b56abfa"
format_stamp: "Formatted at 2025-09-01 20:01:33 on dist-test-slave-2s4d"
server_key: "40ea645ca4dc10a964091ebe3a3006c4"
server_key_iv: "b4e6f015e473716b2268488fc8a7d753"
server_key_version: "encryptionkey@0"
I20250901 20:01:33.638655 17410 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:01:33.640839 17425 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:33.642071 17410 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:33.642179 17410 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "aca98ab52e964dd1bdb2baeb7b56abfa"
format_stamp: "Formatted at 2025-09-01 20:01:33 on dist-test-slave-2s4d"
server_key: "40ea645ca4dc10a964091ebe3a3006c4"
server_key_iv: "b4e6f015e473716b2268488fc8a7d753"
server_key_version: "encryptionkey@0"
I20250901 20:01:33.642274 17410 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:33.669005 17410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:33.671587 17410 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:33.671753 17410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:33.672258 17410 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:33.673048 17410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:33.673095 17410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:33.673153 17410 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:33.673183 17410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:33.681851 17410 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:34789
I20250901 20:01:33.681955 17538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:34789 every 8 connection(s)
I20250901 20:01:33.682778 17410 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:01:33.688350 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 17410
I20250901 20:01:33.688498 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:01:33.688781 14605 external_mini_cluster.cc:1468] Setting key 6ac04e768ef63a834e233494101a2cee
Sep 01 20:01:33 dist-test-slave-2s4d krb5kdc[17026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756893, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:33.694375 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:33.684439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:36255 (local address 127.14.67.126:35409)
0901 20:01:33.684562 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:33.684566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:33.685147 (+ 581us) server_negotiation.cc:407] Connection header received
0901 20:01:33.685887 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:33.685891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:33.685935 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:33.686007 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:33.687162 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:33.687654 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:33.689035 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:33.689438 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:33.691348 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:33.691360 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:33.691363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:33.691385 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:33.692703 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:33.693179 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:33.693183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:33.693184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:33.693321 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:33.693598 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:33.693601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:33.693602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:33.693737 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:33.693809 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:33.694177 (+ 368us) server_negotiation.cc:299] Negotiation successful
0901 20:01:33.694262 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250901 20:01:33.694948 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:33.684649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:35409 (local address 127.14.67.67:36255)
0901 20:01:33.685021 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:33.685056 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:33.685730 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:33.686155 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:33.686162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:33.686536 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:33.687035 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:33.687047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:33.687752 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:33.687754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:33.688835 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:33.688845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:33.689031 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:33.689633 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:33.689674 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:33.691229 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:33.692812 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:33.692816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:33.692824 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:33.693031 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:33.693427 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:33.693432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:33.693433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:33.693513 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:33.693815 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:33.693820 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:33.693996 (+ 176us) client_negotiation.cc:769] Sending connection context
0901 20:01:33.694143 (+ 147us) client_negotiation.cc:240] Negotiation successful
0901 20:01:33.694325 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
I20250901 20:01:33.695825 17539 heartbeater.cc:344] Connected to a master server at 127.14.67.126:35409
I20250901 20:01:33.696017 17539 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:33.696444 17539 heartbeater.cc:507] Master 127.14.67.126:35409 requested a full tablet report, sending...
I20250901 20:01:33.697520 17074 ts_manager.cc:194] Registered new tserver with Master: aca98ab52e964dd1bdb2baeb7b56abfa (127.14.67.67:34789)
I20250901 20:01:33.698151 17074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:36255
I20250901 20:01:33.702317 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250901 20:01:33.713014 17113 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:01:33.711928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.14.67.126:35409)
0901 20:01:33.712057 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:33.712060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:33.712146 (+ 86us) server_negotiation.cc:407] Connection header received
0901 20:01:33.712260 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:33.712262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:33.712302 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:33.712373 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:33.712919 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53834: BlockingRecv error: recv got EOF from 127.0.0.1:53834 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20250901 20:01:33.714442 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 17138
I20250901 20:01:33.719983 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 17274
I20250901 20:01:33.725353 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 17410
I20250901 20:01:33.730381 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 17042
2025-09-01T20:01:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3073 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:01:33 dist-test-slave-2s4d krb5kdc[17551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:01:33 dist-test-slave-2s4d krb5kdc[17551](info): set up 2 sockets
Sep 01 20:01:33 dist-test-slave-2s4d krb5kdc[17551](info): commencing operation
krb5kdc: starting...
W20250901 20:01:35.813264 14605 mini_kdc.cc:121] Time spent starting KDC: real 2.062s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20250901 20:01:36.062981 14605 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.250s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20250901 20:01:36.331947 14605 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.269s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20250901 20:01:36.500802 14605 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.169s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/test-user.keytab.
W20250901 20:01:37.525779 14605 mini_kdc.cc:302] Time spent creating keytab for test-user: real 1.025s user 0.001s sys 0.001s
Sep 01 20:01:37 dist-test-slave-2s4d krb5kdc[17551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:01:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:01:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
W20250901 20:01:38.788416 14605 mini_kdc.cc:279] Time spent creating service keytab for kudu/127.14.67.126: real 1.243s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
W20250901 20:01:39.180671 14605 mini_kdc.cc:279] Time spent creating service keytab for HTTP/127.14.67.126: real 0.392s user 0.000s sys 0.003s
I20250901 20:01:39.181171 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:38531
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:38531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:39.292274 17567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:39.292460 17567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:39.292510 17567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:39.295548 17567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:01:39.295598 17567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:39.295620 17567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:39.295639 17567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:01:39.295660 17567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:01:39.299633 17567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:38531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:38531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:39.300688 17567 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:39.301496 17567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:39.307112 17572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:39.307132 17573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:39.307132 17575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:39.307226 17567 server_base.cc:1047] running on GCE node
I20250901 20:01:39.307763 17567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:39.308650 17567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:39.309809 17567 hybrid_clock.cc:648] HybridClock initialized: now 1756756899309802 us; error 35 us; skew 500 ppm
Sep 01 20:01:39 dist-test-slave-2s4d krb5kdc[17551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:39.312242 17567 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:01:39.313253 17567 webserver.cc:480] Webserver started at http://127.14.67.126:39077/ using document root <none> and password file <none>
I20250901 20:01:39.313807 17567 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:39.313858 17567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:39.314036 17567 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:39.315819 17567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "e0009fa2299644a0883b67c7899dae82"
format_stamp: "Formatted at 2025-09-01 20:01:39 on dist-test-slave-2s4d"
server_key: "829430fc399e70fc0c12dea73cbe705f"
server_key_iv: "8830d6013ea951657b96fe9cef7652bf"
server_key_version: "encryptionkey@0"
I20250901 20:01:39.316280 17567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "e0009fa2299644a0883b67c7899dae82"
format_stamp: "Formatted at 2025-09-01 20:01:39 on dist-test-slave-2s4d"
server_key: "829430fc399e70fc0c12dea73cbe705f"
server_key_iv: "8830d6013ea951657b96fe9cef7652bf"
server_key_version: "encryptionkey@0"
I20250901 20:01:39.319682 17567 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:01:39.321679 17582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:39.322679 17567 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250901 20:01:39.322793 17567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "e0009fa2299644a0883b67c7899dae82"
format_stamp: "Formatted at 2025-09-01 20:01:39 on dist-test-slave-2s4d"
server_key: "829430fc399e70fc0c12dea73cbe705f"
server_key_iv: "8830d6013ea951657b96fe9cef7652bf"
server_key_version: "encryptionkey@0"
I20250901 20:01:39.322875 17567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:39.335114 17567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:39.337610 17567 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:39.337786 17567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:39.345187 17567 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:38531
I20250901 20:01:39.345201 17634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:38531 every 8 connection(s)
I20250901 20:01:39.346164 17567 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:01:39.348690 17635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:39.353998 17635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82: Bootstrap starting.
I20250901 20:01:39.356266 17635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:39.356287 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 17567
I20250901 20:01:39.356382 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:01:39.356604 14605 external_mini_cluster.cc:1468] Setting key a8be1ad613b45ad62638f48d16945a75
I20250901 20:01:39.357045 17635 log.cc:826] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:39.358870 17635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82: No bootstrap required, opened a new log
I20250901 20:01:39.361128 17635 raft_consensus.cc:357] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0009fa2299644a0883b67c7899dae82" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38531 } }
I20250901 20:01:39.361304 17635 raft_consensus.cc:383] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:39.361375 17635 raft_consensus.cc:738] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0009fa2299644a0883b67c7899dae82, State: Initialized, Role: FOLLOWER
I20250901 20:01:39.361923 17635 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0009fa2299644a0883b67c7899dae82" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38531 } }
I20250901 20:01:39.362061 17635 raft_consensus.cc:397] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:01:39.362128 17635 raft_consensus.cc:491] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:01:39.362237 17635 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:39.363124 17635 raft_consensus.cc:513] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0009fa2299644a0883b67c7899dae82" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38531 } }
Sep 01 20:01:39 dist-test-slave-2s4d krb5kdc[17551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:39.363508 17635 leader_election.cc:304] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0009fa2299644a0883b67c7899dae82; no voters:
I20250901 20:01:39.363811 17635 leader_election.cc:290] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:01:39.363893 17640 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:39.364090 17640 raft_consensus.cc:695] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [term 1 LEADER]: Becoming Leader. State: Replica: e0009fa2299644a0883b67c7899dae82, State: Running, Role: LEADER
I20250901 20:01:39.364411 17640 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0009fa2299644a0883b67c7899dae82" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38531 } }
I20250901 20:01:39.364950 17635 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:01:39.365760 17642 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0009fa2299644a0883b67c7899dae82. Latest consensus state: current_term: 1 leader_uuid: "e0009fa2299644a0883b67c7899dae82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0009fa2299644a0883b67c7899dae82" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38531 } } }
I20250901 20:01:39.365964 17642 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:39.366103 17641 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0009fa2299644a0883b67c7899dae82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0009fa2299644a0883b67c7899dae82" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38531 } } }
I20250901 20:01:39.366214 17641 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:39.366415 17649 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:01:39.369768 17649 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:01:39.370364 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:39.357584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.14.67.126:38531)
0901 20:01:39.357970 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:39.357980 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:39.358012 (+ 32us) server_negotiation.cc:407] Connection header received
0901 20:01:39.358561 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:39.358578 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:39.358865 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:39.359168 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:39.360076 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:39.360814 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:39.361448 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:39.361729 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:39.364024 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:39.364045 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:39.364061 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:39.364102 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:39.366244 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:39.366756 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:39.366762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:39.366767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:39.366850 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:39.367128 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:39.367132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:39.367134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:39.367530 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:39.367713 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:39.368349 (+ 636us) server_negotiation.cc:299] Negotiation successful
0901 20:01:39.369637 (+ 1288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":118,"threads_started":1}
I20250901 20:01:39.374701 17649 catalog_manager.cc:1349] Generated new cluster ID: f7769c91f64a4e2ebd7f2f4658435b76
I20250901 20:01:39.374758 17649 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:01:39.385947 17649 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:01:39.387089 17649 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:01:39.397786 17649 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e0009fa2299644a0883b67c7899dae82: Generated new TSK 0
I20250901 20:01:39.398597 17649 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:39.445152 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:38531
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:39.545236 17663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:39.545429 17663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:39.545483 17663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:39.548527 17663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:39.548580 17663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:39.548653 17663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:01:39.552528 17663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:38531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:39.553448 17663 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:39.554204 17663 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:39.560241 17671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:39.560246 17668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:39.560250 17669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:39.560469 17663 server_base.cc:1047] running on GCE node
I20250901 20:01:39.560789 17663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:39.561277 17663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:39.562429 17663 hybrid_clock.cc:648] HybridClock initialized: now 1756756899562415 us; error 38 us; skew 500 ppm
Sep 01 20:01:39 dist-test-slave-2s4d krb5kdc[17551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:39.564867 17663 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:01:39.565840 17663 webserver.cc:480] Webserver started at http://127.14.67.65:44123/ using document root <none> and password file <none>
I20250901 20:01:39.566339 17663 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:39.566391 17663 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:39.566627 17663 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:39.568285 17663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "edecc579a3414ea7ade7048d34594d48"
format_stamp: "Formatted at 2025-09-01 20:01:39 on dist-test-slave-2s4d"
server_key: "3f8e28153d167b017cff831a91cbfbba"
server_key_iv: "1f14219eaf36d0f2ad0c95c106911214"
server_key_version: "encryptionkey@0"
I20250901 20:01:39.568737 17663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "edecc579a3414ea7ade7048d34594d48"
format_stamp: "Formatted at 2025-09-01 20:01:39 on dist-test-slave-2s4d"
server_key: "3f8e28153d167b017cff831a91cbfbba"
server_key_iv: "1f14219eaf36d0f2ad0c95c106911214"
server_key_version: "encryptionkey@0"
I20250901 20:01:39.571789 17663 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:01:39.574023 17678 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:39.574985 17663 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250901 20:01:39.575098 17663 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "edecc579a3414ea7ade7048d34594d48"
format_stamp: "Formatted at 2025-09-01 20:01:39 on dist-test-slave-2s4d"
server_key: "3f8e28153d167b017cff831a91cbfbba"
server_key_iv: "1f14219eaf36d0f2ad0c95c106911214"
server_key_version: "encryptionkey@0"
I20250901 20:01:39.575194 17663 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:39.593261 17663 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:39.595829 17663 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:39.596001 17663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:39.596503 17663 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:39.597267 17663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:39.597318 17663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:39.597374 17663 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:39.597409 17663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:39.605831 17663 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:33119
I20250901 20:01:39.605904 17791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:33119 every 8 connection(s)
I20250901 20:01:39.606729 17663 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:01:39.610165 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 17663
I20250901 20:01:39.610294 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:01:39.610584 14605 external_mini_cluster.cc:1468] Setting key 15a4023f173c512b56d5a930bbe1d190
Sep 01 20:01:39 dist-test-slave-2s4d krb5kdc[17551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
I20250901 20:01:39.619858 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:39.608339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:35353 (local address 127.14.67.126:38531)
0901 20:01:39.608484 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:39.608488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:39.609094 (+ 606us) server_negotiation.cc:407] Connection header received
0901 20:01:39.609867 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:39.609870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:39.609940 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:39.610020 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:39.611996 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:39.612505 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:39.613414 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:39.613649 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:39.615971 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:39.615985 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:39.615987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:39.616008 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:39.617826 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:39.618398 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:39.618401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:39.618402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:39.618444 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:39.618792 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:39.618795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:39.618796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:39.618938 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:39.619017 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:39.619621 (+ 604us) server_negotiation.cc:299] Negotiation successful
0901 20:01:39.619739 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250901 20:01:39.620487 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:39.608547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38531 (local address 127.14.67.65:35353)
0901 20:01:39.608964 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:39.608999 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:39.609678 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:39.610225 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:39.610236 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:39.610793 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:39.611861 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:39.611875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:39.612627 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:39.612634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:39.613265 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:39.613277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:39.613474 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:39.614057 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:39.614078 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:39.615824 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:39.617957 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:39.617963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:39.617974 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:39.618282 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:39.618549 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:39.618554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:39.618559 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:39.618684 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:39.619046 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:39.619052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:39.619328 (+ 276us) client_negotiation.cc:769] Sending connection context
0901 20:01:39.619562 (+ 234us) client_negotiation.cc:240] Negotiation successful
0901 20:01:39.619780 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
I20250901 20:01:39.621647 17792 heartbeater.cc:344] Connected to a master server at 127.14.67.126:38531
I20250901 20:01:39.621886 17792 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:39.622500 17792 heartbeater.cc:507] Master 127.14.67.126:38531 requested a full tablet report, sending...
I20250901 20:01:39.624002 17599 ts_manager.cc:194] Registered new tserver with Master: edecc579a3414ea7ade7048d34594d48 (127.14.67.65:33119)
I20250901 20:01:39.625100 17599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:35353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:39.655097 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:38531
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:39.752393 17799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:39.752581 17799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:39.752636 17799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:39.755606 17799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:39.755678 17799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:39.755792 17799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:01:39.759630 17799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:38531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:39.760528 17799 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:39.761314 17799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:39.767350 17805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:39.767374 17804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:39.767356 17807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:39.767674 17799 server_base.cc:1047] running on GCE node
I20250901 20:01:39.767992 17799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:39.768491 17799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:39.769640 17799 hybrid_clock.cc:648] HybridClock initialized: now 1756756899769622 us; error 41 us; skew 500 ppm
Sep 01 20:01:39 dist-test-slave-2s4d krb5kdc[17551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:39.772079 17799 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:01:39.773005 17799 webserver.cc:480] Webserver started at http://127.14.67.66:40409/ using document root <none> and password file <none>
I20250901 20:01:39.773506 17799 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:39.773557 17799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:39.773727 17799 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:39.775369 17799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "0f468b072a0b45169312bce481b01baa"
format_stamp: "Formatted at 2025-09-01 20:01:39 on dist-test-slave-2s4d"
server_key: "1b232bf2d5d03c3fc90f560a196a5295"
server_key_iv: "4469d62277b529aa137e8c5084f43b4b"
server_key_version: "encryptionkey@0"
I20250901 20:01:39.775820 17799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "0f468b072a0b45169312bce481b01baa"
format_stamp: "Formatted at 2025-09-01 20:01:39 on dist-test-slave-2s4d"
server_key: "1b232bf2d5d03c3fc90f560a196a5295"
server_key_iv: "4469d62277b529aa137e8c5084f43b4b"
server_key_version: "encryptionkey@0"
I20250901 20:01:39.778908 17799 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:01:39.780958 17814 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:39.781864 17799 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:39.781976 17799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "0f468b072a0b45169312bce481b01baa"
format_stamp: "Formatted at 2025-09-01 20:01:39 on dist-test-slave-2s4d"
server_key: "1b232bf2d5d03c3fc90f560a196a5295"
server_key_iv: "4469d62277b529aa137e8c5084f43b4b"
server_key_version: "encryptionkey@0"
I20250901 20:01:39.782053 17799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:39.793738 17799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:39.796294 17799 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:39.796468 17799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:39.796948 17799 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:39.797711 17799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:39.797760 17799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:39.797818 17799 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:39.797847 17799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:39.806104 17799 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:39487
I20250901 20:01:39.806138 17927 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:39487 every 8 connection(s)
I20250901 20:01:39.806995 17799 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:01:39.809417 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 17799
I20250901 20:01:39.809540 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:01:39.809744 14605 external_mini_cluster.cc:1468] Setting key 310901d8fffa1615e3257c20334078bf
Sep 01 20:01:39 dist-test-slave-2s4d krb5kdc[17551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.67.67@KRBTEST.COM; defaulting to no policy
I20250901 20:01:39.818856 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:39.808568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:51695 (local address 127.14.67.126:38531)
0901 20:01:39.808695 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:39.808699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:39.809297 (+ 598us) server_negotiation.cc:407] Connection header received
0901 20:01:39.810183 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:39.810186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:39.810246 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:39.810324 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:39.812309 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:39.812790 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:39.813407 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:39.813590 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:39.815799 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:39.815814 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:39.815816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:39.815839 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:39.817166 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:39.817737 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:39.817740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:39.817741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:39.817781 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:39.818009 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:39.818011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:39.818012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:39.818130 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:39.818195 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:39.818652 (+ 457us) server_negotiation.cc:299] Negotiation successful
0901 20:01:39.818742 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.67@KRBTEST.COM" created.
I20250901 20:01:39.819567 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:39.808783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38531 (local address 127.14.67.66:51695)
0901 20:01:39.809167 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:39.809202 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:39.809999 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:39.810919 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:39.810927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:39.811276 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:39.812147 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:39.812160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:39.812907 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:39.812910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:39.813291 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:39.813297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:39.813475 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:39.814077 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:39.814098 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:39.815635 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:39.817282 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:39.817287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:39.817298 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:39.817641 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:39.817861 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:39.817863 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:39.817865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:39.817942 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:39.818198 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:39.818202 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:39.818425 (+ 223us) client_negotiation.cc:769] Sending connection context
0901 20:01:39.818619 (+ 194us) client_negotiation.cc:240] Negotiation successful
0901 20:01:39.818816 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":105,"threads_started":1}
I20250901 20:01:39.821362 17928 heartbeater.cc:344] Connected to a master server at 127.14.67.126:38531
I20250901 20:01:39.821818 17928 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:39.822650 17928 heartbeater.cc:507] Master 127.14.67.126:38531 requested a full tablet report, sending...
I20250901 20:01:39.823809 17599 ts_manager.cc:194] Registered new tserver with Master: 0f468b072a0b45169312bce481b01baa (127.14.67.66:39487)
I20250901 20:01:39.824565 17599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:51695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:39.861378 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:38531
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:39.963537 17935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:39.963738 17935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:39.963810 17935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:39.967247 17935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:39.967303 17935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:39.967408 17935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:01:39.971370 17935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:38531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:39.972365 17935 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:39.973153 17935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:39.979331 17940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:39.979460 17935 server_base.cc:1047] running on GCE node
W20250901 20:01:39.979332 17943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:39.979595 17941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:39.979913 17935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:39.980424 17935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:39.981562 17935 hybrid_clock.cc:648] HybridClock initialized: now 1756756899981551 us; error 25 us; skew 500 ppm
Sep 01 20:01:39 dist-test-slave-2s4d krb5kdc[17551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:39.984129 17935 init.cc:377] Logged in from keytab as kudu/127.14.67.67@KRBTEST.COM (short username kudu)
I20250901 20:01:39.985131 17935 webserver.cc:480] Webserver started at http://127.14.67.67:35549/ using document root <none> and password file <none>
I20250901 20:01:39.985662 17935 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:39.985710 17935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:39.985888 17935 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:39.987618 17935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "3795330023dd4c3a8bde15096a6a3978"
format_stamp: "Formatted at 2025-09-01 20:01:39 on dist-test-slave-2s4d"
server_key: "b0c2af9f91db4eb25a07730753bce6dc"
server_key_iv: "7c1d3e1b56d5bb539fe80727d7c3bd07"
server_key_version: "encryptionkey@0"
I20250901 20:01:39.988057 17935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "3795330023dd4c3a8bde15096a6a3978"
format_stamp: "Formatted at 2025-09-01 20:01:39 on dist-test-slave-2s4d"
server_key: "b0c2af9f91db4eb25a07730753bce6dc"
server_key_iv: "7c1d3e1b56d5bb539fe80727d7c3bd07"
server_key_version: "encryptionkey@0"
I20250901 20:01:39.991192 17935 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:01:39.993175 17950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:39.994099 17935 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:39.994184 17935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "3795330023dd4c3a8bde15096a6a3978"
format_stamp: "Formatted at 2025-09-01 20:01:39 on dist-test-slave-2s4d"
server_key: "b0c2af9f91db4eb25a07730753bce6dc"
server_key_iv: "7c1d3e1b56d5bb539fe80727d7c3bd07"
server_key_version: "encryptionkey@0"
I20250901 20:01:39.994254 17935 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:40.008168 17935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:40.010788 17935 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:40.010941 17935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:40.011420 17935 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:40.012177 17935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:40.012235 17935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:40.012274 17935 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:40.012295 17935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:40.020877 17935 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:45207
I20250901 20:01:40.020915 18063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:45207 every 8 connection(s)
I20250901 20:01:40.021781 17935 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:01:40.026599 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 17935
I20250901 20:01:40.026716 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:01:40.026935 14605 external_mini_cluster.cc:1468] Setting key 9ae885b5bbf16498702d592d7996ccf6
Sep 01 20:01:40 dist-test-slave-2s4d krb5kdc[17551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:40.032989 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:40.023500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:38475 (local address 127.14.67.126:38531)
0901 20:01:40.023626 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:40.023629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:40.024241 (+ 612us) server_negotiation.cc:407] Connection header received
0901 20:01:40.025001 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:40.025004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:40.025053 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:40.025126 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:40.026761 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:40.027280 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:40.027982 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:40.028170 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:40.030234 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:40.030248 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:40.030250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:40.030274 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:40.031538 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:40.031937 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:40.031939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:40.031940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:40.031975 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:40.032166 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:40.032168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:40.032169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:40.032306 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:40.032375 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:40.032789 (+ 414us) server_negotiation.cc:299] Negotiation successful
0901 20:01:40.032874 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250901 20:01:40.033569 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:40.023711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38531 (local address 127.14.67.67:38475)
0901 20:01:40.024103 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:40.024137 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:40.024832 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:40.025256 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:40.025264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:40.025629 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:40.026594 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:40.026609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:40.027431 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:40.027434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:40.027874 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:40.027886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:40.028076 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:40.028670 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:40.028701 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:40.030110 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:40.031634 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:40.031638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:40.031646 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:40.031858 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:40.032034 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:40.032036 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:40.032038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:40.032106 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:40.032397 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:40.032402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:40.032596 (+ 194us) client_negotiation.cc:769] Sending connection context
0901 20:01:40.032756 (+ 160us) client_negotiation.cc:240] Negotiation successful
0901 20:01:40.032947 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
I20250901 20:01:40.034538 18064 heartbeater.cc:344] Connected to a master server at 127.14.67.126:38531
I20250901 20:01:40.034756 18064 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:40.035216 18064 heartbeater.cc:507] Master 127.14.67.126:38531 requested a full tablet report, sending...
I20250901 20:01:40.036100 17599 ts_manager.cc:194] Registered new tserver with Master: 3795330023dd4c3a8bde15096a6a3978 (127.14.67.67:45207)
I20250901 20:01:40.036594 17599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:38475
I20250901 20:01:40.040278 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250901 20:01:40.163290 17639 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:01:40.160164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.14.67.126:38531)
0901 20:01:40.160302 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:40.160306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:40.161146 (+ 840us) server_negotiation.cc:407] Connection header received
0901 20:01:40.161908 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:40.161911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:40.161964 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:40.162033 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:40.163169 (+ 1136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53520: BlockingRecv error: recv got EOF from 127.0.0.1:53520 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20250901 20:01:40.164507 17639 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:01:40.163847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53536 (local address 127.14.67.126:38531)
0901 20:01:40.163978 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:40.163980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:40.163992 (+ 12us) server_negotiation.cc:407] Connection header received
0901 20:01:40.164072 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:40.164075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:40.164105 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:40.164166 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:40.164411 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53536: BlockingRecv error: recv got EOF from 127.0.0.1:53536 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20250901 20:01:40.165343 17639 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:01:40.164764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.14.67.126:38531)
0901 20:01:40.164855 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:40.164858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:40.164868 (+ 10us) server_negotiation.cc:407] Connection header received
0901 20:01:40.164921 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:40.164923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:40.164952 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:40.165003 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:40.165251 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53550: BlockingRecv error: recv got EOF from 127.0.0.1:53550 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20250901 20:01:40.168951 17639 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:01:40.168280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.14.67.126:38531)
0901 20:01:40.168407 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:40.168413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:40.168426 (+ 13us) server_negotiation.cc:407] Connection header received
0901 20:01:40.168457 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:40.168459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:40.168488 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:40.168552 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:40.168889 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53552: BlockingRecv error: recv got EOF from 127.0.0.1:53552 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20250901 20:01:40.176385 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 17663
I20250901 20:01:40.181622 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 17799
I20250901 20:01:40.186970 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 17935
I20250901 20:01:40.192075 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 17567
2025-09-01T20:01:40Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (6462 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:01:40 dist-test-slave-2s4d krb5kdc[18087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:01:40 dist-test-slave-2s4d krb5kdc[18087](info): set up 2 sockets
Sep 01 20:01:40 dist-test-slave-2s4d krb5kdc[18087](info): commencing operation
krb5kdc: starting...
W20250901 20:01:42.167174 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.954s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:01:42 dist-test-slave-2s4d krb5kdc[18087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:01:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:01:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:42.295258 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:44903
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:44903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:42.397639 18103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:42.397862 18103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:42.397919 18103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:42.401136 18103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:01:42.401199 18103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:42.401222 18103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:42.401245 18103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:01:42.401283 18103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:01:42.405395 18103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:44903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:44903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:42.406424 18103 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:42.407255 18103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:42.412837 18108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:42.412904 18103 server_base.cc:1047] running on GCE node
W20250901 20:01:42.412840 18109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:42.412837 18111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:42.413511 18103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:42.414368 18103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:42.415540 18103 hybrid_clock.cc:648] HybridClock initialized: now 1756756902415527 us; error 34 us; skew 500 ppm
Sep 01 20:01:42 dist-test-slave-2s4d krb5kdc[18087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756902, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:42.418190 18103 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:01:42.419262 18103 webserver.cc:480] Webserver started at http://127.14.67.126:45255/ using document root <none> and password file <none>
I20250901 20:01:42.419798 18103 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:42.419852 18103 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:42.420028 18103 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:42.421696 18103 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "d424498e4ae6473d982dc81dc0303146"
format_stamp: "Formatted at 2025-09-01 20:01:42 on dist-test-slave-2s4d"
server_key: "c97131a56a7a43367c9de28dfab02ab0"
server_key_iv: "1a70166985f8afb3f57ab69c86e75f85"
server_key_version: "encryptionkey@0"
I20250901 20:01:42.422150 18103 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "d424498e4ae6473d982dc81dc0303146"
format_stamp: "Formatted at 2025-09-01 20:01:42 on dist-test-slave-2s4d"
server_key: "c97131a56a7a43367c9de28dfab02ab0"
server_key_iv: "1a70166985f8afb3f57ab69c86e75f85"
server_key_version: "encryptionkey@0"
I20250901 20:01:42.425379 18103 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:01:42.427491 18118 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:42.428444 18103 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:01:42.428550 18103 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "d424498e4ae6473d982dc81dc0303146"
format_stamp: "Formatted at 2025-09-01 20:01:42 on dist-test-slave-2s4d"
server_key: "c97131a56a7a43367c9de28dfab02ab0"
server_key_iv: "1a70166985f8afb3f57ab69c86e75f85"
server_key_version: "encryptionkey@0"
I20250901 20:01:42.428637 18103 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:42.447814 18103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:42.455739 18103 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:42.455925 18103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:42.462987 18103 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:44903
I20250901 20:01:42.462993 18170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:44903 every 8 connection(s)
I20250901 20:01:42.463934 18103 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:01:42.466329 18171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:42.470489 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 18103
I20250901 20:01:42.470644 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:01:42.470904 14605 external_mini_cluster.cc:1468] Setting key e35b1b8f4050691c56b7c8a7d09a009a
I20250901 20:01:42.472419 18171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146: Bootstrap starting.
I20250901 20:01:42.474431 18171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:42.475117 18171 log.cc:826] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:42.476830 18171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146: No bootstrap required, opened a new log
Sep 01 20:01:42 dist-test-slave-2s4d krb5kdc[18087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:42.479571 18171 raft_consensus.cc:357] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d424498e4ae6473d982dc81dc0303146" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44903 } }
I20250901 20:01:42.479782 18171 raft_consensus.cc:383] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:42.479836 18171 raft_consensus.cc:738] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d424498e4ae6473d982dc81dc0303146, State: Initialized, Role: FOLLOWER
I20250901 20:01:42.480228 18171 consensus_queue.cc:260] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d424498e4ae6473d982dc81dc0303146" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44903 } }
I20250901 20:01:42.480332 18171 raft_consensus.cc:397] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:01:42.480403 18171 raft_consensus.cc:491] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:01:42.480518 18171 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:42.481222 18171 raft_consensus.cc:513] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d424498e4ae6473d982dc81dc0303146" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44903 } }
I20250901 20:01:42.481529 18171 leader_election.cc:304] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d424498e4ae6473d982dc81dc0303146; no voters:
I20250901 20:01:42.481786 18171 leader_election.cc:290] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:01:42.481865 18176 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:42.482024 18176 raft_consensus.cc:695] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [term 1 LEADER]: Becoming Leader. State: Replica: d424498e4ae6473d982dc81dc0303146, State: Running, Role: LEADER
I20250901 20:01:42.482301 18176 consensus_queue.cc:237] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d424498e4ae6473d982dc81dc0303146" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44903 } }
I20250901 20:01:42.483021 18171 sys_catalog.cc:564] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:01:42.483314 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:42.472105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.14.67.126:44903)
0901 20:01:42.472576 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:42.472586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:42.472619 (+ 33us) server_negotiation.cc:407] Connection header received
0901 20:01:42.473116 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:42.473131 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:42.473397 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:42.473694 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:42.474602 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:42.475371 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:42.476010 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:42.476286 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:42.478584 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:42.478601 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:42.478611 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:42.478640 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:42.480584 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:42.480994 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:42.480999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:42.481004 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:42.481071 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:42.481280 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:42.481286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:42.481289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:42.481625 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:42.481836 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:42.482238 (+ 402us) server_negotiation.cc:299] Negotiation successful
0901 20:01:42.482492 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":146,"threads_started":1}
I20250901 20:01:42.484481 18178 sys_catalog.cc:455] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d424498e4ae6473d982dc81dc0303146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d424498e4ae6473d982dc81dc0303146" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44903 } } }
I20250901 20:01:42.484581 18178 sys_catalog.cc:458] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:42.484815 18177 sys_catalog.cc:455] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d424498e4ae6473d982dc81dc0303146. Latest consensus state: current_term: 1 leader_uuid: "d424498e4ae6473d982dc81dc0303146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d424498e4ae6473d982dc81dc0303146" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44903 } } }
I20250901 20:01:42.484905 18177 sys_catalog.cc:458] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:42.485165 18185 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:01:42.486433 14605 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250901 20:01:42.488140 18185 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:01:42.493268 18185 catalog_manager.cc:1349] Generated new cluster ID: 9bbeb1a65ce34a4db6d45eb517851876
I20250901 20:01:42.493337 18185 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:01:42.512885 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:42.508336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.14.67.126:44903)
0901 20:01:42.508465 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:42.508470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:42.508630 (+ 160us) server_negotiation.cc:407] Connection header received
0901 20:01:42.508763 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:42.508767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:42.508824 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:42.508927 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:42.509631 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:42.510374 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:42.511173 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:42.511372 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:42.511860 (+ 488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:42.511880 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:42.511883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:42.512433 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:42.512466 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:42.512485 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:42.512578 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:42.512662 (+ 84us) server_negotiation.cc:299] Negotiation successful
0901 20:01:42.512734 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250901 20:01:42.521881 18185 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:01:42.522842 18185 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:01:42.529006 18185 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d424498e4ae6473d982dc81dc0303146: Generated new TSK 0
I20250901 20:01:42.529665 18185 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 20:01:42.536167 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 18103
2025-09-01T20:01:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2341 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:01:42 dist-test-slave-2s4d krb5kdc[18204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:01:42 dist-test-slave-2s4d krb5kdc[18204](info): set up 2 sockets
Sep 01 20:01:42 dist-test-slave-2s4d krb5kdc[18204](info): commencing operation
krb5kdc: starting...
W20250901 20:01:44.543452 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:01:44 dist-test-slave-2s4d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:01:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:01:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:44.671728 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:35383
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:35383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:44.771442 18220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:44.771639 18220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:44.771723 18220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:44.774806 18220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:01:44.774861 18220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:44.774902 18220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:44.774942 18220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:01:44.774981 18220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:01:44.778916 18220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:35383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:35383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:44.779951 18220 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:44.780751 18220 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:44.786126 18228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:44.786149 18225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:44.786149 18226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:44.786465 18220 server_base.cc:1047] running on GCE node
I20250901 20:01:44.787159 18220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:44.788060 18220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:44.789244 18220 hybrid_clock.cc:648] HybridClock initialized: now 1756756904789230 us; error 33 us; skew 500 ppm
Sep 01 20:01:44 dist-test-slave-2s4d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:44.791787 18220 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:01:44.792788 18220 webserver.cc:480] Webserver started at http://127.14.67.126:43619/ using document root <none> and password file <none>
I20250901 20:01:44.793296 18220 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:44.793372 18220 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:44.793552 18220 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:44.795301 18220 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "820693956d5a4d5fabeae76eb15428e2"
format_stamp: "Formatted at 2025-09-01 20:01:44 on dist-test-slave-2s4d"
server_key: "7825ea5c1329778d9655f907214f73e2"
server_key_iv: "e58ef002cc933e863529d560e080afd9"
server_key_version: "encryptionkey@0"
I20250901 20:01:44.795790 18220 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "820693956d5a4d5fabeae76eb15428e2"
format_stamp: "Formatted at 2025-09-01 20:01:44 on dist-test-slave-2s4d"
server_key: "7825ea5c1329778d9655f907214f73e2"
server_key_iv: "e58ef002cc933e863529d560e080afd9"
server_key_version: "encryptionkey@0"
I20250901 20:01:44.799474 18220 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250901 20:01:44.801725 18235 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:44.802863 18220 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20250901 20:01:44.802984 18220 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "820693956d5a4d5fabeae76eb15428e2"
format_stamp: "Formatted at 2025-09-01 20:01:44 on dist-test-slave-2s4d"
server_key: "7825ea5c1329778d9655f907214f73e2"
server_key_iv: "e58ef002cc933e863529d560e080afd9"
server_key_version: "encryptionkey@0"
I20250901 20:01:44.803081 18220 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:44.833917 18220 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:44.836719 18220 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:44.836871 18220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:44.843662 18287 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:35383 every 8 connection(s)
I20250901 20:01:44.843667 18220 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:35383
I20250901 20:01:44.844625 18220 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:01:44.846912 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 18220
I20250901 20:01:44.847043 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:01:44.847129 18288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:44.847311 14605 external_mini_cluster.cc:1468] Setting key 520fc07639035da7bc7fd32d0b6559c8
I20250901 20:01:44.852517 18288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2: Bootstrap starting.
Sep 01 20:01:44 dist-test-slave-2s4d krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:44.855202 18288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:44.856040 18288 log.cc:826] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:44.857823 18288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2: No bootstrap required, opened a new log
I20250901 20:01:44.859320 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:44.848543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.14.67.126:35383)
0901 20:01:44.848944 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:44.848954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:44.848986 (+ 32us) server_negotiation.cc:407] Connection header received
0901 20:01:44.849510 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:44.849525 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:44.849802 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:44.850087 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:44.851014 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:44.851765 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:44.852402 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:44.852703 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:44.855017 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:44.855033 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:44.855047 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:44.855090 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:44.856910 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:44.857359 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:44.857363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:44.857367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:44.857442 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:44.857720 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:44.857724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:44.857725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:44.858079 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:44.858244 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:44.858499 (+ 255us) server_negotiation.cc:299] Negotiation successful
0901 20:01:44.858783 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":132,"threads_started":1}
I20250901 20:01:44.860297 18288 raft_consensus.cc:357] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820693956d5a4d5fabeae76eb15428e2" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35383 } }
I20250901 20:01:44.860473 18288 raft_consensus.cc:383] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:44.860512 18288 raft_consensus.cc:738] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820693956d5a4d5fabeae76eb15428e2, State: Initialized, Role: FOLLOWER
I20250901 20:01:44.860898 18288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820693956d5a4d5fabeae76eb15428e2" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35383 } }
I20250901 20:01:44.861020 18288 raft_consensus.cc:397] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:01:44.861079 18288 raft_consensus.cc:491] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:01:44.861198 18288 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:44.861967 18288 raft_consensus.cc:513] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820693956d5a4d5fabeae76eb15428e2" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35383 } }
I20250901 20:01:44.862262 18288 leader_election.cc:304] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 820693956d5a4d5fabeae76eb15428e2; no voters:
I20250901 20:01:44.862534 18288 leader_election.cc:290] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:01:44.862685 18293 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:44.862862 18293 raft_consensus.cc:695] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [term 1 LEADER]: Becoming Leader. State: Replica: 820693956d5a4d5fabeae76eb15428e2, State: Running, Role: LEADER
I20250901 20:01:44.863127 18293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820693956d5a4d5fabeae76eb15428e2" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35383 } }
I20250901 20:01:44.863547 18288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:01:44.865366 18294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "820693956d5a4d5fabeae76eb15428e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820693956d5a4d5fabeae76eb15428e2" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35383 } } }
I20250901 20:01:44.865483 18294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:44.865434 18295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 820693956d5a4d5fabeae76eb15428e2. Latest consensus state: current_term: 1 leader_uuid: "820693956d5a4d5fabeae76eb15428e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820693956d5a4d5fabeae76eb15428e2" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35383 } } }
I20250901 20:01:44.865535 18295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:44.865965 18303 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:01:44.868849 18303 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:01:44.873912 18303 catalog_manager.cc:1349] Generated new cluster ID: 01ee66d2fd5a43d4b1dda98be2983dcc
I20250901 20:01:44.873979 18303 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:01:44.890973 18303 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:01:44.892133 18303 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:01:44.902884 18303 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 820693956d5a4d5fabeae76eb15428e2: Generated new TSK 0
I20250901 20:01:44.903529 18303 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:44.955363 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:35383
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:45.082531 18316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:45.082731 18316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:45.082788 18316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:45.085803 18316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:45.085856 18316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:45.085930 18316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:01:45.090015 18316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:35383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:45.090948 18316 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:45.091696 18316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:45.097729 18324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:45.097801 18322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:45.097803 18321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:45.097926 18316 server_base.cc:1047] running on GCE node
I20250901 20:01:45.098318 18316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:45.098843 18316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:45.100028 18316 hybrid_clock.cc:648] HybridClock initialized: now 1756756905099998 us; error 43 us; skew 500 ppm
Sep 01 20:01:45 dist-test-slave-2s4d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:45.102991 18316 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:01:45.103893 18316 webserver.cc:480] Webserver started at http://127.14.67.65:40757/ using document root <none> and password file <none>
I20250901 20:01:45.104367 18316 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:45.104421 18316 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:45.104561 18316 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:45.106122 18316 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "2fba1dd8e65e41ddb4d90ff1b17a79ba"
format_stamp: "Formatted at 2025-09-01 20:01:45 on dist-test-slave-2s4d"
server_key: "466f0c665b241f0892a8db7d2880c09a"
server_key_iv: "d90cbf3e59409788c187c926a9d62858"
server_key_version: "encryptionkey@0"
I20250901 20:01:45.106582 18316 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "2fba1dd8e65e41ddb4d90ff1b17a79ba"
format_stamp: "Formatted at 2025-09-01 20:01:45 on dist-test-slave-2s4d"
server_key: "466f0c665b241f0892a8db7d2880c09a"
server_key_iv: "d90cbf3e59409788c187c926a9d62858"
server_key_version: "encryptionkey@0"
I20250901 20:01:45.109745 18316 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250901 20:01:45.111768 18331 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:45.112686 18316 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:01:45.112779 18316 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "2fba1dd8e65e41ddb4d90ff1b17a79ba"
format_stamp: "Formatted at 2025-09-01 20:01:45 on dist-test-slave-2s4d"
server_key: "466f0c665b241f0892a8db7d2880c09a"
server_key_iv: "d90cbf3e59409788c187c926a9d62858"
server_key_version: "encryptionkey@0"
I20250901 20:01:45.112874 18316 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:45.132019 18316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:45.134590 18316 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:45.134755 18316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:45.135239 18316 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:45.136006 18316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:45.136050 18316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:45.136096 18316 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:45.136125 18316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:45.144778 18316 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:36553
I20250901 20:01:45.144812 18444 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:36553 every 8 connection(s)
I20250901 20:01:45.145635 18316 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:01:45.151393 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 18316
I20250901 20:01:45.151512 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:01:45.151795 14605 external_mini_cluster.cc:1468] Setting key 6c45264c710e3522b882f15702aaeab0
Sep 01 20:01:45 dist-test-slave-2s4d krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:45.157784 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:45.147227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:40977 (local address 127.14.67.126:35383)
0901 20:01:45.147356 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:45.147359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:45.147955 (+ 596us) server_negotiation.cc:407] Connection header received
0901 20:01:45.148737 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:45.148741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:45.148800 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:45.148874 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:45.150376 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.150929 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:45.151718 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.151884 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:45.154029 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:45.154043 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:45.154045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:45.154066 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:45.155817 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:45.156254 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:45.156260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:45.156265 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:45.156320 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:45.156587 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:45.156592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:45.156595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:45.156799 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:45.156905 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:45.157398 (+ 493us) server_negotiation.cc:299] Negotiation successful
0901 20:01:45.157539 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250901 20:01:45.158186 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:45.147443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:35383 (local address 127.14.67.65:40977)
0901 20:01:45.147821 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:45.147856 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:45.148554 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:45.149025 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:45.149032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:45.149369 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:45.150229 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:45.150241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.151047 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:45.151050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:45.151595 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:45.151607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.151794 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:45.152394 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:45.152425 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:45.153887 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:45.155934 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:45.155938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:45.155945 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:45.156158 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:45.156419 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:45.156422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:45.156423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:45.156500 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:45.156928 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:45.156933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:45.157150 (+ 217us) client_negotiation.cc:769] Sending connection context
0901 20:01:45.157321 (+ 171us) client_negotiation.cc:240] Negotiation successful
0901 20:01:45.157531 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":96,"threads_started":1}
I20250901 20:01:45.159300 18445 heartbeater.cc:344] Connected to a master server at 127.14.67.126:35383
I20250901 20:01:45.159552 18445 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:45.160166 18445 heartbeater.cc:507] Master 127.14.67.126:35383 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
I20250901 20:01:45.161943 18252 ts_manager.cc:194] Registered new tserver with Master: 2fba1dd8e65e41ddb4d90ff1b17a79ba (127.14.67.65:36553)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
I20250901 20:01:45.163224 18252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:40977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:45.195461 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:35383
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:45.296712 18452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:45.296916 18452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:45.297006 18452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:45.300170 18452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:45.300228 18452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:45.300333 18452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:01:45.304304 18452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:35383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:45.305244 18452 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:45.306018 18452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:45.312314 18460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:45.312325 18457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:45.312325 18458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:45.312749 18452 server_base.cc:1047] running on GCE node
I20250901 20:01:45.313335 18452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:45.314070 18452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:45.315304 18452 hybrid_clock.cc:648] HybridClock initialized: now 1756756905315280 us; error 38 us; skew 500 ppm
Sep 01 20:01:45 dist-test-slave-2s4d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:45.318670 18452 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:01:45.319782 18452 webserver.cc:480] Webserver started at http://127.14.67.66:37069/ using document root <none> and password file <none>
I20250901 20:01:45.320302 18452 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:45.320353 18452 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:45.320510 18452 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:45.322185 18452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "c3b017b7e81341fc9cd152c9575e2f48"
format_stamp: "Formatted at 2025-09-01 20:01:45 on dist-test-slave-2s4d"
server_key: "a89dd85fd2ca9cbafe695210d4000faf"
server_key_iv: "3658ba04fd1067f105bc3af179173b0d"
server_key_version: "encryptionkey@0"
I20250901 20:01:45.322674 18452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "c3b017b7e81341fc9cd152c9575e2f48"
format_stamp: "Formatted at 2025-09-01 20:01:45 on dist-test-slave-2s4d"
server_key: "a89dd85fd2ca9cbafe695210d4000faf"
server_key_iv: "3658ba04fd1067f105bc3af179173b0d"
server_key_version: "encryptionkey@0"
I20250901 20:01:45.326164 18452 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250901 20:01:45.328542 18467 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:45.329939 18452 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:45.330068 18452 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "c3b017b7e81341fc9cd152c9575e2f48"
format_stamp: "Formatted at 2025-09-01 20:01:45 on dist-test-slave-2s4d"
server_key: "a89dd85fd2ca9cbafe695210d4000faf"
server_key_iv: "3658ba04fd1067f105bc3af179173b0d"
server_key_version: "encryptionkey@0"
I20250901 20:01:45.330200 18452 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:45.348701 18452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:45.352389 18452 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:45.352584 18452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:45.353144 18452 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:45.353987 18452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:45.354036 18452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:45.354075 18452 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:45.354091 18452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:45.363719 18452 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:38009
I20250901 20:01:45.363742 18580 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:38009 every 8 connection(s)
I20250901 20:01:45.364668 18452 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:01:45.370455 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 18452
I20250901 20:01:45.370594 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:01:45.370967 14605 external_mini_cluster.cc:1468] Setting key 82b7f275f8e0b690d443783afe2a2585
Sep 01 20:01:45 dist-test-slave-2s4d krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:45.379676 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:45.366993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:36777 (local address 127.14.67.126:35383)
0901 20:01:45.367135 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:45.367139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:45.367935 (+ 796us) server_negotiation.cc:407] Connection header received
0901 20:01:45.369004 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:45.369007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:45.369056 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:45.369130 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:45.370911 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.371553 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:45.372620 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.372790 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:45.375377 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:45.375398 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:45.375401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:45.375431 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:45.377239 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:45.377925 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:45.377928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:45.377930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:45.377977 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:45.378370 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:45.378374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:45.378376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:45.378652 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:45.378761 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:45.379437 (+ 676us) server_negotiation.cc:299] Negotiation successful
0901 20:01:45.379554 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250901 20:01:45.380550 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:45.367266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:35383 (local address 127.14.67.66:36777)
0901 20:01:45.367698 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:45.367736 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:45.368773 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:45.369273 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:45.369281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:45.369674 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:45.370696 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:45.370713 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.371690 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:45.371694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:45.372441 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:45.372455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.372730 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:45.373510 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:45.373536 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:45.375174 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:45.377389 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:45.377398 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:45.377413 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:45.377779 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:45.378088 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:45.378093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:45.378097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:45.378215 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:45.378776 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:45.378783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:45.379099 (+ 316us) client_negotiation.cc:769] Sending connection context
0901 20:01:45.379362 (+ 263us) client_negotiation.cc:240] Negotiation successful
0901 20:01:45.379670 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.14.67.67@KRBTEST.COM; defaulting to no policy
I20250901 20:01:45.381767 18581 heartbeater.cc:344] Connected to a master server at 127.14.67.126:35383
I20250901 20:01:45.382112 18581 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.67@KRBTEST.COM" created.
I20250901 20:01:45.383075 18581 heartbeater.cc:507] Master 127.14.67.126:35383 requested a full tablet report, sending...
I20250901 20:01:45.384334 18252 ts_manager.cc:194] Registered new tserver with Master: c3b017b7e81341fc9cd152c9575e2f48 (127.14.67.66:38009)
I20250901 20:01:45.384922 18252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:36777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:45.417425 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:35383
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:45.518756 18588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:45.518945 18588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:45.519002 18588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:45.522017 18588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:45.522068 18588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:45.522192 18588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:01:45.526046 18588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:35383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:45.527011 18588 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:45.527791 18588 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:45.533779 18594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:45.533784 18593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:45.533788 18596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:45.534037 18588 server_base.cc:1047] running on GCE node
I20250901 20:01:45.534368 18588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:45.534992 18588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:45.536154 18588 hybrid_clock.cc:648] HybridClock initialized: now 1756756905536125 us; error 46 us; skew 500 ppm
Sep 01 20:01:45 dist-test-slave-2s4d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:45.538956 18588 init.cc:377] Logged in from keytab as kudu/127.14.67.67@KRBTEST.COM (short username kudu)
I20250901 20:01:45.539876 18588 webserver.cc:480] Webserver started at http://127.14.67.67:40511/ using document root <none> and password file <none>
I20250901 20:01:45.540392 18588 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:45.540455 18588 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:45.540620 18588 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:45.542273 18588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "76b448747d184bb4b69a714dc2d88abe"
format_stamp: "Formatted at 2025-09-01 20:01:45 on dist-test-slave-2s4d"
server_key: "c2a254340c6d8b72f8321eacc194c687"
server_key_iv: "4c74b5f2b0eda5ede759b78d635bf244"
server_key_version: "encryptionkey@0"
I20250901 20:01:45.542730 18588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "76b448747d184bb4b69a714dc2d88abe"
format_stamp: "Formatted at 2025-09-01 20:01:45 on dist-test-slave-2s4d"
server_key: "c2a254340c6d8b72f8321eacc194c687"
server_key_iv: "4c74b5f2b0eda5ede759b78d635bf244"
server_key_version: "encryptionkey@0"
I20250901 20:01:45.545806 18588 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20250901 20:01:45.547811 18603 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:45.548808 18588 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:01:45.548910 18588 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "76b448747d184bb4b69a714dc2d88abe"
format_stamp: "Formatted at 2025-09-01 20:01:45 on dist-test-slave-2s4d"
server_key: "c2a254340c6d8b72f8321eacc194c687"
server_key_iv: "4c74b5f2b0eda5ede759b78d635bf244"
server_key_version: "encryptionkey@0"
I20250901 20:01:45.548998 18588 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:45.565550 18588 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:45.568112 18588 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:45.568282 18588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:45.568773 18588 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:45.569538 18588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:45.569588 18588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:45.569653 18588 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:45.569685 18588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:45.577946 18588 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:44101
I20250901 20:01:45.577972 18716 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:44101 every 8 connection(s)
I20250901 20:01:45.578836 18588 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:01:45.582394 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 18588
I20250901 20:01:45.582499 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:01:45.582758 14605 external_mini_cluster.cc:1468] Setting key e8887e1e2647a158d2183486ebbeecad
Sep 01 20:01:45 dist-test-slave-2s4d krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:45.590749 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:45.580399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:48737 (local address 127.14.67.126:35383)
0901 20:01:45.580554 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:45.580557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:45.581123 (+ 566us) server_negotiation.cc:407] Connection header received
0901 20:01:45.581968 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:45.581971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:45.582016 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:45.582089 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:45.583797 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.584323 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:45.585008 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.585153 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:45.587295 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:45.587308 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:45.587311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:45.587333 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:45.588695 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:45.589562 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:45.589565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:45.589566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:45.589610 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:45.589866 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:45.589869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:45.589870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:45.590020 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:45.590097 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:45.590550 (+ 453us) server_negotiation.cc:299] Negotiation successful
0901 20:01:45.590639 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20250901 20:01:45.591391 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:45.580602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:35383 (local address 127.14.67.67:48737)
0901 20:01:45.580995 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:45.581027 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:45.581791 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:45.582213 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:45.582219 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:45.582674 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:45.583666 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:45.583680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.584462 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:45.584465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:45.584878 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:45.584886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.585030 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:45.585553 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:45.585574 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:45.587159 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:45.588935 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:45.588950 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:45.588964 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:45.589450 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:45.589690 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:45.589693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:45.589695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:45.589771 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:45.590117 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:45.590122 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:45.590309 (+ 187us) client_negotiation.cc:769] Sending connection context
0901 20:01:45.590483 (+ 174us) client_negotiation.cc:240] Negotiation successful
0901 20:01:45.590722 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":97,"threads_started":1}
I20250901 20:01:45.592410 18717 heartbeater.cc:344] Connected to a master server at 127.14.67.126:35383
I20250901 20:01:45.592618 18717 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:45.593075 18717 heartbeater.cc:507] Master 127.14.67.126:35383 requested a full tablet report, sending...
I20250901 20:01:45.593945 18252 ts_manager.cc:194] Registered new tserver with Master: 76b448747d184bb4b69a714dc2d88abe (127.14.67.67:44101)
I20250901 20:01:45.594450 18252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:48737
I20250901 20:01:45.596756 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 01 20:01:45 dist-test-slave-2s4d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756905, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 01 20:01:45 dist-test-slave-2s4d krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756905, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:45.618461 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:45.611343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.14.67.126:35383)
0901 20:01:45.611486 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:45.611489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:45.611585 (+ 96us) server_negotiation.cc:407] Connection header received
0901 20:01:45.611681 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:45.611684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:45.611723 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:45.611792 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:45.612529 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.612987 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:45.613561 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:45.613676 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:45.615749 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:45.615763 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:45.615765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:45.615786 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:45.617142 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:45.617497 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:45.617500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:45.617502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:45.617541 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:45.617741 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:45.617744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:45.617746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:45.617896 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:45.618016 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:45.618202 (+ 186us) server_negotiation.cc:299] Negotiation successful
0901 20:01:45.618308 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20250901 20:01:45.618793 18252 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39744
I20250901 20:01:45.619472 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 18316
I20250901 20:01:45.625031 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 18452
I20250901 20:01:45.630439 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 18588
I20250901 20:01:45.635464 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 18220
2025-09-01T20:01:45Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3101 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:01:45 dist-test-slave-2s4d krb5kdc[18729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:01:45 dist-test-slave-2s4d krb5kdc[18729](info): set up 2 sockets
Sep 01 20:01:45 dist-test-slave-2s4d krb5kdc[18729](info): commencing operation
krb5kdc: starting...
W20250901 20:01:47.635201 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:01:47 dist-test-slave-2s4d krb5kdc[18729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:01:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:01:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:47.762113 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:38875
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:38875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:47.863636 18745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:47.863826 18745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:47.863878 18745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:47.866859 18745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:01:47.866909 18745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:47.866930 18745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:47.866948 18745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:01:47.866966 18745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:01:47.870935 18745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:38875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:38875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:47.871824 18745 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:47.872586 18745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:47.877822 18751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:47.877836 18750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:47.877836 18753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:47.878213 18745 server_base.cc:1047] running on GCE node
I20250901 20:01:47.878706 18745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:47.879529 18745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:47.880687 18745 hybrid_clock.cc:648] HybridClock initialized: now 1756756907880666 us; error 35 us; skew 500 ppm
Sep 01 20:01:47 dist-test-slave-2s4d krb5kdc[18729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756907, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:47.883647 18745 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:01:47.884723 18745 webserver.cc:480] Webserver started at http://127.14.67.126:39143/ using document root <none> and password file <none>
I20250901 20:01:47.885237 18745 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:47.885293 18745 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:47.885488 18745 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:47.887156 18745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "702efafc01494fa895b5e8c6f73f8922"
format_stamp: "Formatted at 2025-09-01 20:01:47 on dist-test-slave-2s4d"
server_key: "ce99ca600857917d5af0c04cc7a94a14"
server_key_iv: "feaa5101b4c5c03608f2e8a52b1e15c9"
server_key_version: "encryptionkey@0"
I20250901 20:01:47.887631 18745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "702efafc01494fa895b5e8c6f73f8922"
format_stamp: "Formatted at 2025-09-01 20:01:47 on dist-test-slave-2s4d"
server_key: "ce99ca600857917d5af0c04cc7a94a14"
server_key_iv: "feaa5101b4c5c03608f2e8a52b1e15c9"
server_key_version: "encryptionkey@0"
I20250901 20:01:47.891026 18745 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250901 20:01:47.893322 18760 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:47.894218 18745 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250901 20:01:47.894330 18745 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "702efafc01494fa895b5e8c6f73f8922"
format_stamp: "Formatted at 2025-09-01 20:01:47 on dist-test-slave-2s4d"
server_key: "ce99ca600857917d5af0c04cc7a94a14"
server_key_iv: "feaa5101b4c5c03608f2e8a52b1e15c9"
server_key_version: "encryptionkey@0"
I20250901 20:01:47.894415 18745 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:47.903053 18745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:47.909878 18745 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:47.910058 18745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:47.916733 18745 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:38875
I20250901 20:01:47.916755 18812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:38875 every 8 connection(s)
I20250901 20:01:47.917671 18745 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:01:47.920116 18813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:47.925168 18813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922: Bootstrap starting.
I20250901 20:01:47.927004 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 18745
I20250901 20:01:47.927126 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:01:47.927349 14605 external_mini_cluster.cc:1468] Setting key e4b3e04a227dbb5770daea66ed83603e
I20250901 20:01:47.927532 18813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:47.928349 18813 log.cc:826] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:47.930112 18813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922: No bootstrap required, opened a new log
I20250901 20:01:47.932840 18813 raft_consensus.cc:357] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702efafc01494fa895b5e8c6f73f8922" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38875 } }
I20250901 20:01:47.933033 18813 raft_consensus.cc:383] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:47.933095 18813 raft_consensus.cc:738] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 702efafc01494fa895b5e8c6f73f8922, State: Initialized, Role: FOLLOWER
I20250901 20:01:47.933574 18813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702efafc01494fa895b5e8c6f73f8922" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38875 } }
I20250901 20:01:47.933749 18813 raft_consensus.cc:397] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:01:47.933811 18813 raft_consensus.cc:491] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:01:47.933915 18813 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [term 0 FOLLOWER]: Advancing to term 1
Sep 01 20:01:47 dist-test-slave-2s4d krb5kdc[18729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:47.934798 18813 raft_consensus.cc:513] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702efafc01494fa895b5e8c6f73f8922" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38875 } }
I20250901 20:01:47.935097 18813 leader_election.cc:304] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 702efafc01494fa895b5e8c6f73f8922; no voters:
I20250901 20:01:47.935351 18813 leader_election.cc:290] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:01:47.935448 18818 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:47.935586 18818 raft_consensus.cc:695] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [term 1 LEADER]: Becoming Leader. State: Replica: 702efafc01494fa895b5e8c6f73f8922, State: Running, Role: LEADER
I20250901 20:01:47.935832 18818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702efafc01494fa895b5e8c6f73f8922" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38875 } }
I20250901 20:01:47.936416 18813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:01:47.936928 18819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "702efafc01494fa895b5e8c6f73f8922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702efafc01494fa895b5e8c6f73f8922" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38875 } } }
I20250901 20:01:47.937050 18819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:47.936935 18818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 702efafc01494fa895b5e8c6f73f8922. Latest consensus state: current_term: 1 leader_uuid: "702efafc01494fa895b5e8c6f73f8922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702efafc01494fa895b5e8c6f73f8922" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38875 } } }
I20250901 20:01:47.937323 18818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:47.937654 18823 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:01:47.940357 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:47.928394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.14.67.126:38875)
0901 20:01:47.928766 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:47.928775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:47.928807 (+ 32us) server_negotiation.cc:407] Connection header received
0901 20:01:47.929285 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:47.929302 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:47.929563 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:47.929805 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:47.930835 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:47.931584 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:47.932482 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:47.932729 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:47.935147 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:47.935165 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:47.935175 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:47.935204 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:47.937902 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:47.938374 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:47.938379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:47.938384 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:47.938485 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:47.938746 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:47.938751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:47.938753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:47.939095 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:47.939275 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:47.939567 (+ 292us) server_negotiation.cc:299] Negotiation successful
0901 20:01:47.939832 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":120,"threads_started":1}
I20250901 20:01:47.940971 18823 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:01:47.946126 18823 catalog_manager.cc:1349] Generated new cluster ID: 542264aa634a4e7e8110f68f548d58b3
I20250901 20:01:47.946197 18823 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:01:47.960953 18823 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:01:47.962037 18823 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:01:47.979192 18823 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 702efafc01494fa895b5e8c6f73f8922: Generated new TSK 0
I20250901 20:01:47.980230 18823 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:48.027606 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:38875
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:48.132740 18841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:48.132937 18841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:48.132997 18841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:48.136065 18841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:48.136121 18841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:48.136230 18841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:01:48.140286 18841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:38875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:48.141264 18841 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:48.142067 18841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:48.148190 18849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:48.148252 18847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:48.148196 18846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:48.148556 18841 server_base.cc:1047] running on GCE node
I20250901 20:01:48.148881 18841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:48.149353 18841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:48.150529 18841 hybrid_clock.cc:648] HybridClock initialized: now 1756756908150491 us; error 27 us; skew 500 ppm
Sep 01 20:01:48 dist-test-slave-2s4d krb5kdc[18729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:48.153271 18841 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:01:48.154157 18841 webserver.cc:480] Webserver started at http://127.14.67.65:37867/ using document root <none> and password file <none>
I20250901 20:01:48.154703 18841 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:48.154754 18841 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:48.154899 18841 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:48.156454 18841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "4383efd594954a5e8142171f72994edf"
format_stamp: "Formatted at 2025-09-01 20:01:48 on dist-test-slave-2s4d"
server_key: "47e8812612930342f7f36b5cdb19f6e9"
server_key_iv: "c079401abfa02cdff320d179b8118cfe"
server_key_version: "encryptionkey@0"
I20250901 20:01:48.156862 18841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "4383efd594954a5e8142171f72994edf"
format_stamp: "Formatted at 2025-09-01 20:01:48 on dist-test-slave-2s4d"
server_key: "47e8812612930342f7f36b5cdb19f6e9"
server_key_iv: "c079401abfa02cdff320d179b8118cfe"
server_key_version: "encryptionkey@0"
I20250901 20:01:48.159897 18841 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:01:48.161832 18856 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:48.162804 18841 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:48.162890 18841 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "4383efd594954a5e8142171f72994edf"
format_stamp: "Formatted at 2025-09-01 20:01:48 on dist-test-slave-2s4d"
server_key: "47e8812612930342f7f36b5cdb19f6e9"
server_key_iv: "c079401abfa02cdff320d179b8118cfe"
server_key_version: "encryptionkey@0"
I20250901 20:01:48.162983 18841 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:48.175943 18841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:48.178449 18841 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:48.178642 18841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:48.179126 18841 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:48.179881 18841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:48.179926 18841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:48.179966 18841 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:48.180001 18841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:48.188169 18841 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:40643
I20250901 20:01:48.188206 18969 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:40643 every 8 connection(s)
I20250901 20:01:48.189039 18841 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:01:48.192484 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 18841
I20250901 20:01:48.192587 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:01:48.192816 14605 external_mini_cluster.cc:1468] Setting key 6dc2ab0c38b92968ddd94176f133dcc3
Sep 01 20:01:48 dist-test-slave-2s4d krb5kdc[18729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:48.201203 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:48.190649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:52495 (local address 127.14.67.126:38875)
0901 20:01:48.190769 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:48.190773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:48.191356 (+ 583us) server_negotiation.cc:407] Connection header received
0901 20:01:48.192093 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:48.192095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:48.192153 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:48.192225 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:48.193424 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.194156 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:48.195758 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.195991 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:48.198095 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:48.198108 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:48.198111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:48.198132 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:48.199525 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:48.199969 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:48.199972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:48.199974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:48.200011 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:48.200251 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:48.200255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:48.200257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:48.200404 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:48.200490 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:48.200928 (+ 438us) server_negotiation.cc:299] Negotiation successful
0901 20:01:48.201046 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
I20250901 20:01:48.201754 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:48.190861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38875 (local address 127.14.67.65:52495)
0901 20:01:48.191229 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:48.191262 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:48.191931 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:48.192361 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:48.192368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:48.192789 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:48.193322 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:48.193333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.194290 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:48.194295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:48.195604 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:48.195618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.195817 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:48.196411 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:48.196432 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:48.197969 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:48.199644 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:48.199649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:48.199659 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:48.199871 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:48.200090 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:48.200095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:48.200098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:48.200169 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:48.200500 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:48.200508 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:48.200710 (+ 202us) client_negotiation.cc:769] Sending connection context
0901 20:01:48.200881 (+ 171us) client_negotiation.cc:240] Negotiation successful
0901 20:01:48.201119 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":95,"threads_started":1}
I20250901 20:01:48.202788 18970 heartbeater.cc:344] Connected to a master server at 127.14.67.126:38875
I20250901 20:01:48.202992 18970 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:48.203455 18970 heartbeater.cc:507] Master 127.14.67.126:38875 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
I20250901 20:01:48.204955 18777 ts_manager.cc:194] Registered new tserver with Master: 4383efd594954a5e8142171f72994edf (127.14.67.65:40643)
I20250901 20:01:48.206086 18777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:52495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:48.235365 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:38875
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:48.335258 18977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:48.335457 18977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:48.335512 18977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:48.338630 18977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:48.338685 18977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:48.338759 18977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:01:48.342674 18977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:38875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:48.343578 18977 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:48.344345 18977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:48.350759 18983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:48.350768 18982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:48.350852 18977 server_base.cc:1047] running on GCE node
W20250901 20:01:48.350765 18985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:48.351325 18977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:48.351846 18977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:48.353008 18977 hybrid_clock.cc:648] HybridClock initialized: now 1756756908353002 us; error 29 us; skew 500 ppm
Sep 01 20:01:48 dist-test-slave-2s4d krb5kdc[18729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:48.355934 18977 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:01:48.356940 18977 webserver.cc:480] Webserver started at http://127.14.67.66:36133/ using document root <none> and password file <none>
I20250901 20:01:48.357461 18977 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:48.357514 18977 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:48.357695 18977 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:48.359385 18977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "9b50e05d076b4ef0a31f6ed89cef9948"
format_stamp: "Formatted at 2025-09-01 20:01:48 on dist-test-slave-2s4d"
server_key: "65625b8275b4fbf5ec6cf64ce291b327"
server_key_iv: "b4c34ae46b97b4f9837c0443592927e1"
server_key_version: "encryptionkey@0"
I20250901 20:01:48.359841 18977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "9b50e05d076b4ef0a31f6ed89cef9948"
format_stamp: "Formatted at 2025-09-01 20:01:48 on dist-test-slave-2s4d"
server_key: "65625b8275b4fbf5ec6cf64ce291b327"
server_key_iv: "b4c34ae46b97b4f9837c0443592927e1"
server_key_version: "encryptionkey@0"
I20250901 20:01:48.363008 18977 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:01:48.365041 18992 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:48.366466 18977 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:48.366604 18977 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "9b50e05d076b4ef0a31f6ed89cef9948"
format_stamp: "Formatted at 2025-09-01 20:01:48 on dist-test-slave-2s4d"
server_key: "65625b8275b4fbf5ec6cf64ce291b327"
server_key_iv: "b4c34ae46b97b4f9837c0443592927e1"
server_key_version: "encryptionkey@0"
I20250901 20:01:48.366696 18977 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:48.377192 18977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:48.379814 18977 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:48.379982 18977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:48.380492 18977 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:48.381278 18977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:48.381327 18977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:48.381397 18977 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:48.381423 18977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:48.390369 18977 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:45085
I20250901 20:01:48.390404 19105 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:45085 every 8 connection(s)
I20250901 20:01:48.391279 18977 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
Sep 01 20:01:48 dist-test-slave-2s4d krb5kdc[18729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:48.400352 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 18977
I20250901 20:01:48.400492 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:01:48.400738 14605 external_mini_cluster.cc:1468] Setting key 4f4871a85f9ed1dfc646dc66c8bb990d
I20250901 20:01:48.402863 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:48.392877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:46693 (local address 127.14.67.126:38875)
0901 20:01:48.393051 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:48.393056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:48.393646 (+ 590us) server_negotiation.cc:407] Connection header received
0901 20:01:48.394361 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:48.394364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:48.394414 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:48.394488 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:48.395652 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.396158 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:48.397228 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.397360 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:48.399382 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:48.399398 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:48.399400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:48.399423 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:48.400930 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:48.401424 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:48.401427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:48.401429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:48.401480 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:48.401793 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:48.401799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:48.401804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:48.401983 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:48.402085 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:48.402657 (+ 572us) server_negotiation.cc:299] Negotiation successful
0901 20:01:48.402752 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20250901 20:01:48.403754 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:48.393114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38875 (local address 127.14.67.66:46693)
0901 20:01:48.393497 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:48.393532 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:48.394195 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:48.394652 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:48.394659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:48.395014 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:48.395532 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:48.395544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.396258 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:48.396261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:48.397121 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:48.397130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.397276 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:48.397809 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:48.397829 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:48.399248 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:48.401031 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:48.401036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:48.401049 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:48.401309 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:48.401595 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:48.401597 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:48.401599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:48.401705 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:48.402091 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:48.402097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:48.402378 (+ 281us) client_negotiation.cc:769] Sending connection context
0901 20:01:48.402627 (+ 249us) client_negotiation.cc:240] Negotiation successful
0901 20:01:48.402911 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":96,"threads_started":1}
I20250901 20:01:48.404893 19106 heartbeater.cc:344] Connected to a master server at 127.14.67.126:38875
I20250901 20:01:48.405104 19106 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:48.405592 19106 heartbeater.cc:507] Master 127.14.67.126:38875 requested a full tablet report, sending...
I20250901 20:01:48.406607 18777 ts_manager.cc:194] Registered new tserver with Master: 9b50e05d076b4ef0a31f6ed89cef9948 (127.14.67.66:45085)
I20250901 20:01:48.407418 18777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:46693
WARNING: no policy specified for kudu/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:48.444783 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:38875
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:48.548195 19113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:48.548389 19113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:48.548444 19113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:48.551736 19113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:48.551790 19113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:48.551867 19113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:01:48.556674 19113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:38875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:48.557703 19113 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:48.558481 19113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:48.564580 19119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:48.564590 19121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:48.564610 19118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:48.564924 19113 server_base.cc:1047] running on GCE node
I20250901 20:01:48.565225 19113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:48.565711 19113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:48.566879 19113 hybrid_clock.cc:648] HybridClock initialized: now 1756756908566859 us; error 45 us; skew 500 ppm
Sep 01 20:01:48 dist-test-slave-2s4d krb5kdc[18729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:48.569639 19113 init.cc:377] Logged in from keytab as kudu/127.14.67.67@KRBTEST.COM (short username kudu)
I20250901 20:01:48.570647 19113 webserver.cc:480] Webserver started at http://127.14.67.67:41635/ using document root <none> and password file <none>
I20250901 20:01:48.571260 19113 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:48.571323 19113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:48.571501 19113 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:48.573143 19113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "9792bdbed4b549f7b8cb1f4009999efb"
format_stamp: "Formatted at 2025-09-01 20:01:48 on dist-test-slave-2s4d"
server_key: "52a265dfb7343f7896d7fa571f01a408"
server_key_iv: "6a99ae1ecc3e1349d562f8b26fe76b14"
server_key_version: "encryptionkey@0"
I20250901 20:01:48.573585 19113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "9792bdbed4b549f7b8cb1f4009999efb"
format_stamp: "Formatted at 2025-09-01 20:01:48 on dist-test-slave-2s4d"
server_key: "52a265dfb7343f7896d7fa571f01a408"
server_key_iv: "6a99ae1ecc3e1349d562f8b26fe76b14"
server_key_version: "encryptionkey@0"
I20250901 20:01:48.576973 19113 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:01:48.578944 19128 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:48.579893 19113 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:48.579990 19113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "9792bdbed4b549f7b8cb1f4009999efb"
format_stamp: "Formatted at 2025-09-01 20:01:48 on dist-test-slave-2s4d"
server_key: "52a265dfb7343f7896d7fa571f01a408"
server_key_iv: "6a99ae1ecc3e1349d562f8b26fe76b14"
server_key_version: "encryptionkey@0"
I20250901 20:01:48.580080 19113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:48.593683 19113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:48.596554 19113 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:48.596730 19113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:48.597213 19113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:48.598035 19113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:48.598096 19113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:48.598148 19113 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:48.598174 19113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:48.606671 19113 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:41463
I20250901 20:01:48.606714 19241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:41463 every 8 connection(s)
I20250901 20:01:48.607579 19113 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:01:48.609692 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 19113
I20250901 20:01:48.609810 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:01:48.610095 14605 external_mini_cluster.cc:1468] Setting key 78884ff59d1e1552bcfdd07d352b8e22
Sep 01 20:01:48 dist-test-slave-2s4d krb5kdc[18729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:48.619407 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:48.609183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:48997 (local address 127.14.67.126:38875)
0901 20:01:48.609304 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:48.609307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:48.610005 (+ 698us) server_negotiation.cc:407] Connection header received
0901 20:01:48.610996 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:48.611000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:48.611050 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:48.611118 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:48.612406 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.612896 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:48.613997 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.614122 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:48.616092 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:48.616106 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:48.616108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:48.616132 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:48.617448 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:48.618152 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:48.618155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:48.618156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:48.618200 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:48.618560 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:48.618564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:48.618566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:48.618711 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:48.618790 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:48.619200 (+ 410us) server_negotiation.cc:299] Negotiation successful
0901 20:01:48.619293 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250901 20:01:48.619961 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:48.609373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38875 (local address 127.14.67.67:48997)
0901 20:01:48.609855 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:48.609901 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:48.610799 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:48.611248 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:48.611255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:48.611704 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:48.612284 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:48.612297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.613030 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:48.613034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:48.613890 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:48.613899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.614047 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:48.614601 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:48.614628 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:48.615966 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:48.617636 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:48.617645 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:48.617658 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:48.618005 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:48.618336 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:48.618340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:48.618342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:48.618438 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:48.618795 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:48.618799 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:48.618989 (+ 190us) client_negotiation.cc:769] Sending connection context
0901 20:01:48.619147 (+ 158us) client_negotiation.cc:240] Negotiation successful
0901 20:01:48.619348 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20250901 20:01:48.620925 19242 heartbeater.cc:344] Connected to a master server at 127.14.67.126:38875
I20250901 20:01:48.621220 19242 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:48.621682 19242 heartbeater.cc:507] Master 127.14.67.126:38875 requested a full tablet report, sending...
I20250901 20:01:48.622669 18777 ts_manager.cc:194] Registered new tserver with Master: 9792bdbed4b549f7b8cb1f4009999efb (127.14.67.67:41463)
I20250901 20:01:48.623355 18777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:48997
I20250901 20:01:48.623844 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Sep 01 20:01:48 dist-test-slave-2s4d krb5kdc[18729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 01 20:01:48 dist-test-slave-2s4d krb5kdc[18729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.65@KRBTEST.COM
I20250901 20:01:48.648144 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:48.639573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.14.67.65:40643)
0901 20:01:48.639879 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:48.639886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:48.639906 (+ 20us) server_negotiation.cc:407] Connection header received
0901 20:01:48.639952 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:48.639957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:48.640103 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:48.640324 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:48.641010 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.641693 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:48.642380 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.642665 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:48.645031 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:48.645054 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:48.645065 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:48.645089 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:48.646743 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:48.647120 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:48.647124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:48.647125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:48.647166 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:48.647381 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:48.647383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:48.647384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:48.647539 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:48.647688 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:48.647812 (+ 124us) server_negotiation.cc:299] Negotiation successful
0901 20:01:48.647931 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":124,"threads_started":1}
Sep 01 20:01:48 dist-test-slave-2s4d krb5kdc[18729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:48.658576 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:48.650635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.14.67.126:38875)
0901 20:01:48.650767 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:48.650771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:48.650783 (+ 12us) server_negotiation.cc:407] Connection header received
0901 20:01:48.650867 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:48.650869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:48.650909 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:48.650973 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:48.651788 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.652370 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:48.653141 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:48.653307 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:48.655465 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:48.655483 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:48.655485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:48.655508 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:48.656977 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:48.657363 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:48.657366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:48.657367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:48.657418 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:48.657670 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:48.657673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:48.657674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:48.657897 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:48.658020 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:48.658210 (+ 190us) server_negotiation.cc:299] Negotiation successful
0901 20:01:48.658379 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20250901 20:01:48.658932 18776 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59822
I20250901 20:01:48.659549 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 18841
I20250901 20:01:48.664831 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 18977
I20250901 20:01:48.670476 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 19113
I20250901 20:01:48.675684 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 18745
2025-09-01T20:01:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3041 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:01:48 dist-test-slave-2s4d krb5kdc[19254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:01:48 dist-test-slave-2s4d krb5kdc[19254](info): set up 2 sockets
Sep 01 20:01:48 dist-test-slave-2s4d krb5kdc[19254](info): commencing operation
krb5kdc: starting...
W20250901 20:01:50.683301 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.985s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:01:50 dist-test-slave-2s4d krb5kdc[19254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:01:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:01:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:50.817790 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:44849
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:33699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:44849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:50.924062 19270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:50.924276 19270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:50.924331 19270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:50.928041 19270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:01:50.928117 19270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:50.928141 19270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:50.928161 19270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:01:50.928179 19270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:01:50.932600 19270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:33699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:44849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:44849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:50.933573 19270 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:50.934365 19270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:50.939692 19276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:50.939653 19278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:50.939648 19275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:50.940047 19270 server_base.cc:1047] running on GCE node
I20250901 20:01:50.940497 19270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:50.941448 19270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:50.942579 19270 hybrid_clock.cc:648] HybridClock initialized: now 1756756910942574 us; error 35 us; skew 500 ppm
Sep 01 20:01:50 dist-test-slave-2s4d krb5kdc[19254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756910, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:50.945231 19270 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:01:50.945920 19270 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:50.945982 19270 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:50.946158 19270 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:50.947878 19270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "29dd0cc2c10740d38855a6a1c3618d4f"
format_stamp: "Formatted at 2025-09-01 20:01:50 on dist-test-slave-2s4d"
server_key: "e0cb64743fd4bc5a5bb2bd7f81c60089"
server_key_iv: "b89f532000d4a9a8eee503163d9271c6"
server_key_version: "encryptionkey@0"
I20250901 20:01:50.948335 19270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "29dd0cc2c10740d38855a6a1c3618d4f"
format_stamp: "Formatted at 2025-09-01 20:01:50 on dist-test-slave-2s4d"
server_key: "e0cb64743fd4bc5a5bb2bd7f81c60089"
server_key_iv: "b89f532000d4a9a8eee503163d9271c6"
server_key_version: "encryptionkey@0"
I20250901 20:01:50.951525 19270 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250901 20:01:50.953656 19284 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:50.954694 19270 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:50.954794 19270 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "29dd0cc2c10740d38855a6a1c3618d4f"
format_stamp: "Formatted at 2025-09-01 20:01:50 on dist-test-slave-2s4d"
server_key: "e0cb64743fd4bc5a5bb2bd7f81c60089"
server_key_iv: "b89f532000d4a9a8eee503163d9271c6"
server_key_version: "encryptionkey@0"
I20250901 20:01:50.954876 19270 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:50.976053 19270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:50.979552 19270 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:50.979743 19270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:50.988340 19270 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:44849
I20250901 20:01:50.988340 19336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:44849 every 8 connection(s)
I20250901 20:01:50.989188 19270 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:01:50.992496 19337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:50.993338 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 19270
I20250901 20:01:50.993461 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:01:50.993732 14605 external_mini_cluster.cc:1468] Setting key cae14e5e15fe967071989755abec2aa3
I20250901 20:01:50.998936 19337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f: Bootstrap starting.
Sep 01 20:01:50 dist-test-slave-2s4d krb5kdc[19254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:51.001387 19337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:51.002152 19337 log.cc:826] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:51.003911 19337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f: No bootstrap required, opened a new log
I20250901 20:01:51.006007 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:50.995094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.14.67.126:44849)
0901 20:01:50.995522 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:50.995532 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:50.995564 (+ 32us) server_negotiation.cc:407] Connection header received
0901 20:01:50.996090 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:50.996105 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:50.996383 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:50.996671 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:50.997584 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:50.998318 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:50.999015 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:50.999294 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.001525 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:51.001545 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:51.001556 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:51.001586 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:51.003723 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:51.004147 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:51.004152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:51.004156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:51.004223 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:51.004452 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:51.004455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:51.004457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:51.004764 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:51.004940 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:51.005216 (+ 276us) server_negotiation.cc:299] Negotiation successful
0901 20:01:51.005453 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":133,"threads_started":1}
I20250901 20:01:51.006748 19337 raft_consensus.cc:357] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dd0cc2c10740d38855a6a1c3618d4f" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44849 } }
I20250901 20:01:51.006933 19337 raft_consensus.cc:383] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:51.006991 19337 raft_consensus.cc:738] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29dd0cc2c10740d38855a6a1c3618d4f, State: Initialized, Role: FOLLOWER
I20250901 20:01:51.007427 19337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dd0cc2c10740d38855a6a1c3618d4f" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44849 } }
I20250901 20:01:51.007573 19337 raft_consensus.cc:397] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:01:51.007638 19337 raft_consensus.cc:491] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:01:51.007762 19337 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:51.008620 19337 raft_consensus.cc:513] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dd0cc2c10740d38855a6a1c3618d4f" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44849 } }
I20250901 20:01:51.008908 19337 leader_election.cc:304] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29dd0cc2c10740d38855a6a1c3618d4f; no voters:
I20250901 20:01:51.009179 19337 leader_election.cc:290] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:01:51.009250 19342 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:51.009502 19342 raft_consensus.cc:695] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [term 1 LEADER]: Becoming Leader. State: Replica: 29dd0cc2c10740d38855a6a1c3618d4f, State: Running, Role: LEADER
I20250901 20:01:51.010002 19342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dd0cc2c10740d38855a6a1c3618d4f" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44849 } }
I20250901 20:01:51.010855 19337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:01:51.012032 19343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29dd0cc2c10740d38855a6a1c3618d4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dd0cc2c10740d38855a6a1c3618d4f" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44849 } } }
I20250901 20:01:51.012180 19343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:51.012643 19344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29dd0cc2c10740d38855a6a1c3618d4f. Latest consensus state: current_term: 1 leader_uuid: "29dd0cc2c10740d38855a6a1c3618d4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dd0cc2c10740d38855a6a1c3618d4f" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44849 } } }
I20250901 20:01:51.012817 19344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:51.012820 19349 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:01:51.016050 19349 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:01:51.020889 19349 catalog_manager.cc:1349] Generated new cluster ID: ee8744dfef8047c890547fd37b114f67
I20250901 20:01:51.020956 19349 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:01:51.033381 19349 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:01:51.034624 19349 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:01:51.047029 19349 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f: Generated new TSK 0
I20250901 20:01:51.047827 19349 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:51.093910 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:44849
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:33699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:51.197455 19365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:51.197671 19365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:51.197847 19365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:51.201784 19365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:51.201861 19365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:51.201953 19365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:01:51.206029 19365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:33699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:44849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:51.207000 19365 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:51.207814 19365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:51.213718 19370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:51.213809 19365 server_base.cc:1047] running on GCE node
W20250901 20:01:51.213730 19373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:51.213730 19371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:51.214226 19365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:51.214764 19365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:51.215915 19365 hybrid_clock.cc:648] HybridClock initialized: now 1756756911215890 us; error 41 us; skew 500 ppm
Sep 01 20:01:51 dist-test-slave-2s4d krb5kdc[19254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756911, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:51.218539 19365 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:01:51.219198 19365 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:51.219259 19365 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:51.219431 19365 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:51.221050 19365 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "100b65d5640645f88e73f36d8d3e7516"
format_stamp: "Formatted at 2025-09-01 20:01:51 on dist-test-slave-2s4d"
server_key: "8350c052926d08d5476b613ccef56231"
server_key_iv: "1ee265b357605c1e76a60f9c9c5866a9"
server_key_version: "encryptionkey@0"
I20250901 20:01:51.221545 19365 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "100b65d5640645f88e73f36d8d3e7516"
format_stamp: "Formatted at 2025-09-01 20:01:51 on dist-test-slave-2s4d"
server_key: "8350c052926d08d5476b613ccef56231"
server_key_iv: "1ee265b357605c1e76a60f9c9c5866a9"
server_key_version: "encryptionkey@0"
I20250901 20:01:51.224922 19365 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250901 20:01:51.227061 19379 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:51.228250 19365 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:51.228367 19365 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "100b65d5640645f88e73f36d8d3e7516"
format_stamp: "Formatted at 2025-09-01 20:01:51 on dist-test-slave-2s4d"
server_key: "8350c052926d08d5476b613ccef56231"
server_key_iv: "1ee265b357605c1e76a60f9c9c5866a9"
server_key_version: "encryptionkey@0"
I20250901 20:01:51.228466 19365 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:51.251401 19365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:51.253958 19365 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:51.254137 19365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:51.254695 19365 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:51.255499 19365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:51.255548 19365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:51.255609 19365 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:51.255643 19365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:51.263999 19365 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:39071
I20250901 20:01:51.264096 19492 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:39071 every 8 connection(s)
I20250901 20:01:51.264592 19365 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:01:51.269618 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 19365
I20250901 20:01:51.269726 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:01:51.269973 14605 external_mini_cluster.cc:1468] Setting key a97aea78b84722ff6d414b16e4df481b
Sep 01 20:01:51 dist-test-slave-2s4d krb5kdc[19254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756911, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:51.276304 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.266097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:55723 (local address 127.14.67.126:44849)
0901 20:01:51.266229 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:51.266233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:51.266837 (+ 604us) server_negotiation.cc:407] Connection header received
0901 20:01:51.267631 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:51.267634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:51.267694 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:51.267771 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:51.269317 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.269865 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.270500 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.270688 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.273049 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:51.273063 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:51.273066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:51.273088 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:51.274558 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:51.275020 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:51.275023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:51.275025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:51.275069 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:51.275320 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:51.275323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:51.275325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:51.275477 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:51.275565 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:51.276023 (+ 458us) server_negotiation.cc:299] Negotiation successful
0901 20:01:51.276141 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250901 20:01:51.276878 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.266301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:44849 (local address 127.14.67.65:55723)
0901 20:01:51.266702 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:51.266736 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:51.267443 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:51.267930 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:51.267937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:51.268294 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:51.269165 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.269178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.269971 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.269977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:51.270389 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.270396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.270608 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.271226 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:51.271260 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:51.272893 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:51.274677 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:51.274682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:51.274693 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:51.274912 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:51.275154 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:51.275157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:51.275159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:51.275236 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:51.275589 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:51.275595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:51.275801 (+ 206us) client_negotiation.cc:769] Sending connection context
0901 20:01:51.275981 (+ 180us) client_negotiation.cc:240] Negotiation successful
0901 20:01:51.276231 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20250901 20:01:51.277925 19493 heartbeater.cc:344] Connected to a master server at 127.14.67.126:44849
I20250901 20:01:51.278148 19493 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
I20250901 20:01:51.278664 19493 heartbeater.cc:507] Master 127.14.67.126:44849 requested a full tablet report, sending...
I20250901 20:01:51.280030 19301 ts_manager.cc:194] Registered new tserver with Master: 100b65d5640645f88e73f36d8d3e7516 (127.14.67.65:39071)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
I20250901 20:01:51.281437 19301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:55723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:51.315618 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:44849
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:33699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:51.419955 19500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:51.420153 19500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:51.420210 19500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:51.423337 19500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:51.423393 19500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:51.423472 19500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:01:51.427461 19500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:33699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:44849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:51.428382 19500 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:51.429128 19500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:51.435071 19505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:51.435112 19508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:51.435281 19500 server_base.cc:1047] running on GCE node
W20250901 20:01:51.435112 19506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:51.435786 19500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:51.436280 19500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:51.437426 19500 hybrid_clock.cc:648] HybridClock initialized: now 1756756911437410 us; error 29 us; skew 500 ppm
Sep 01 20:01:51 dist-test-slave-2s4d krb5kdc[19254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756911, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:51.440220 19500 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:01:51.440835 19500 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:51.440887 19500 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:51.441056 19500 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:51.442736 19500 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "59aaf7d9556d446d857b3edbc7b4d9eb"
format_stamp: "Formatted at 2025-09-01 20:01:51 on dist-test-slave-2s4d"
server_key: "34ce03056d2d0788ad6e6f408dbb68a4"
server_key_iv: "d9d71f15d3dea390b8685e6d338c773b"
server_key_version: "encryptionkey@0"
I20250901 20:01:51.443181 19500 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "59aaf7d9556d446d857b3edbc7b4d9eb"
format_stamp: "Formatted at 2025-09-01 20:01:51 on dist-test-slave-2s4d"
server_key: "34ce03056d2d0788ad6e6f408dbb68a4"
server_key_iv: "d9d71f15d3dea390b8685e6d338c773b"
server_key_version: "encryptionkey@0"
I20250901 20:01:51.446180 19500 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250901 20:01:51.448171 19514 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:51.449043 19500 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:01:51.449128 19500 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "59aaf7d9556d446d857b3edbc7b4d9eb"
format_stamp: "Formatted at 2025-09-01 20:01:51 on dist-test-slave-2s4d"
server_key: "34ce03056d2d0788ad6e6f408dbb68a4"
server_key_iv: "d9d71f15d3dea390b8685e6d338c773b"
server_key_version: "encryptionkey@0"
I20250901 20:01:51.449220 19500 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:51.481765 19500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:51.484652 19500 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:51.484807 19500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:51.485283 19500 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:51.486044 19500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:51.486088 19500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:51.486126 19500 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:51.486143 19500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:51.494304 19500 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:35979
I20250901 20:01:51.494349 19627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:35979 every 8 connection(s)
I20250901 20:01:51.494844 19500 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:01:51.502007 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 19500
I20250901 20:01:51.502144 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:01:51.502384 14605 external_mini_cluster.cc:1468] Setting key 1ee4292f47072da28744456aa791428e
Sep 01 20:01:51 dist-test-slave-2s4d krb5kdc[19254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756911, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:51.507041 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.496351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:55697 (local address 127.14.67.126:44849)
0901 20:01:51.496510 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:51.496515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:51.497124 (+ 609us) server_negotiation.cc:407] Connection header received
0901 20:01:51.497893 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:51.497896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:51.497946 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:51.498017 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:51.499732 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.500234 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.500864 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.500996 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.503237 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:51.503254 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:51.503257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:51.503280 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:51.504806 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:51.505445 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:51.505452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:51.505455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:51.505512 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:51.505866 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:51.505872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:51.505877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:51.506063 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:51.506164 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:51.506675 (+ 511us) server_negotiation.cc:299] Negotiation successful
0901 20:01:51.506811 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250901 20:01:51.507550 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.496585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:44849 (local address 127.14.67.66:55697)
0901 20:01:51.496988 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:51.497023 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:51.497717 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:51.498146 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:51.498153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:51.498620 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:51.499597 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.499611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.500348 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.500351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:51.500750 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.500758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.500900 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.501479 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:51.501501 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:51.503090 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:51.504924 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:51.504932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:51.504951 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:51.505307 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:51.505622 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:51.505626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:51.505630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:51.505758 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:51.506164 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:51.506169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:51.506377 (+ 208us) client_negotiation.cc:769] Sending connection context
0901 20:01:51.506618 (+ 241us) client_negotiation.cc:240] Negotiation successful
0901 20:01:51.506829 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":91,"threads_started":1}
I20250901 20:01:51.508790 19628 heartbeater.cc:344] Connected to a master server at 127.14.67.126:44849
I20250901 20:01:51.509027 19628 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:51.509509 19628 heartbeater.cc:507] Master 127.14.67.126:44849 requested a full tablet report, sending...
I20250901 20:01:51.510475 19301 ts_manager.cc:194] Registered new tserver with Master: 59aaf7d9556d446d857b3edbc7b4d9eb (127.14.67.66:35979)
WARNING: no policy specified for kudu/127.14.67.67@KRBTEST.COM; defaulting to no policy
I20250901 20:01:51.511042 19301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:55697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:01:51.545389 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:44849
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:33699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:01:51.646623 19635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:51.646827 19635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:51.646896 19635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:51.650023 19635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:51.650082 19635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:51.650199 19635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:01:51.654310 19635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:33699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:44849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:51.655295 19635 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:51.656105 19635 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:51.661996 19641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:51.662051 19640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:51.662134 19635 server_base.cc:1047] running on GCE node
W20250901 20:01:51.661994 19643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:51.662688 19635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:51.663199 19635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:51.664353 19635 hybrid_clock.cc:648] HybridClock initialized: now 1756756911664327 us; error 35 us; skew 500 ppm
Sep 01 20:01:51 dist-test-slave-2s4d krb5kdc[19254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756911, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:01:51.667032 19635 init.cc:377] Logged in from keytab as kudu/127.14.67.67@KRBTEST.COM (short username kudu)
I20250901 20:01:51.667704 19635 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:51.667770 19635 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:51.667938 19635 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:51.669548 19635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "d26c8cc255114587839f28b6ca4cef94"
format_stamp: "Formatted at 2025-09-01 20:01:51 on dist-test-slave-2s4d"
server_key: "2d66fd58cbd4a846ea2bb5b127c89408"
server_key_iv: "d344de89ee7361deee1ed7e44e841d2d"
server_key_version: "encryptionkey@0"
I20250901 20:01:51.670006 19635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "d26c8cc255114587839f28b6ca4cef94"
format_stamp: "Formatted at 2025-09-01 20:01:51 on dist-test-slave-2s4d"
server_key: "2d66fd58cbd4a846ea2bb5b127c89408"
server_key_iv: "d344de89ee7361deee1ed7e44e841d2d"
server_key_version: "encryptionkey@0"
I20250901 20:01:51.673063 19635 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:01:51.675087 19649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:51.676007 19635 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250901 20:01:51.676103 19635 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "d26c8cc255114587839f28b6ca4cef94"
format_stamp: "Formatted at 2025-09-01 20:01:51 on dist-test-slave-2s4d"
server_key: "2d66fd58cbd4a846ea2bb5b127c89408"
server_key_iv: "d344de89ee7361deee1ed7e44e841d2d"
server_key_version: "encryptionkey@0"
I20250901 20:01:51.676187 19635 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:51.695699 19635 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:51.698571 19635 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:51.698758 19635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:51.699301 19635 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:51.700099 19635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:51.700146 19635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:51.700206 19635 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:51.700240 19635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:51.708957 19635 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:40829
I20250901 20:01:51.708985 19762 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:40829 every 8 connection(s)
I20250901 20:01:51.709473 19635 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:01:51.710480 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 19635
I20250901 20:01:51.710640 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:01:51.710899 14605 external_mini_cluster.cc:1468] Setting key 074cd772e1fe826cc0019f9b0de2be22
Sep 01 20:01:51 dist-test-slave-2s4d krb5kdc[19254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756911, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:01:51.721122 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.711262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:58471 (local address 127.14.67.126:44849)
0901 20:01:51.711431 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:51.711436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:51.712067 (+ 631us) server_negotiation.cc:407] Connection header received
0901 20:01:51.712973 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:51.712976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:51.713021 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:51.713104 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:51.714747 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.715247 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.715845 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.715976 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.718046 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:51.718061 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:51.718063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:51.718087 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:51.719399 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:51.719854 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:51.719858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:51.719860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:51.719900 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:51.720149 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:51.720152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:51.720154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:51.720294 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:51.720404 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:51.720850 (+ 446us) server_negotiation.cc:299] Negotiation successful
0901 20:01:51.720962 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20250901 20:01:51.721717 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.711521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:44849 (local address 127.14.67.67:58471)
0901 20:01:51.711937 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:51.711980 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:51.712808 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:51.713247 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:51.713255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:51.713650 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:51.714617 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.714632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.715350 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.715353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:51.715742 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.715749 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.715898 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.716452 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:01:51.716474 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:51.717903 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:51.719509 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:51.719515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:51.719525 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:51.719759 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:51.719987 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:01:51.719990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:01:51.719991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:01:51.720067 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:01:51.720403 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:51.720410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:51.720628 (+ 218us) client_negotiation.cc:769] Sending connection context
0901 20:01:51.720806 (+ 178us) client_negotiation.cc:240] Negotiation successful
0901 20:01:51.721058 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
I20250901 20:01:51.722694 19763 heartbeater.cc:344] Connected to a master server at 127.14.67.126:44849
I20250901 20:01:51.722932 19763 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:51.723407 19763 heartbeater.cc:507] Master 127.14.67.126:44849 requested a full tablet report, sending...
I20250901 20:01:51.724368 19301 ts_manager.cc:194] Registered new tserver with Master: d26c8cc255114587839f28b6ca4cef94 (127.14.67.67:40829)
I20250901 20:01:51.724888 19301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:58471
I20250901 20:01:51.736181 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250901 20:01:51.744472 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.737936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48832 (local address 127.14.67.126:44849)
0901 20:01:51.738093 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:51.738098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:51.738245 (+ 147us) server_negotiation.cc:407] Connection header received
0901 20:01:51.738376 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:51.738379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:51.738429 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:51.738497 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:51.739322 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.739898 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.740504 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.740716 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.741683 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:51.741701 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:51.741703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:01:51.741725 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:51.743059 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:51.743453 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:51.743456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:51.743457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:51.743497 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:01:51.743704 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:01:51.743708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:01:51.743710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:01:51.743869 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:51.743991 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:51.744205 (+ 214us) server_negotiation.cc:299] Negotiation successful
0901 20:01:51.744318 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20250901 20:01:51.746953 19301 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 20:01:51.749239 19301 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:01:51.762581 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.758202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:40829 (local address 127.0.0.1:35958)
0901 20:01:51.758941 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:51.758954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:51.759033 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:51.759551 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:51.759554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:51.759577 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:51.759883 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.759892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.761201 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.761205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:51.762005 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.762013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.762123 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.762178 (+ 55us) client_negotiation.cc:769] Sending connection context
0901 20:01:51.762281 (+ 103us) client_negotiation.cc:240] Negotiation successful
0901 20:01:51.762380 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":90,"threads_started":1}
I20250901 20:01:51.763069 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.758668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35958 (local address 127.14.67.67:40829)
0901 20:01:51.759041 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:51.759047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:51.759108 (+ 61us) server_negotiation.cc:407] Connection header received
0901 20:01:51.759187 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:51.759195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:51.759342 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:51.759531 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:51.760024 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.761075 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.762126 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.762695 (+ 569us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.762793 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:51.762861 (+ 68us) server_negotiation.cc:299] Negotiation successful
0901 20:01:51.762941 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":99,"threads_started":1}
I20250901 20:01:51.763145 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.757531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52366 (local address 127.14.67.65:39071)
0901 20:01:51.758142 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:51.758150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:51.758639 (+ 489us) server_negotiation.cc:407] Connection header received
0901 20:01:51.758937 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:51.758943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:51.759149 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:51.759285 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:51.760013 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.760827 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.761959 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.762479 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.762640 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:51.762880 (+ 240us) server_negotiation.cc:299] Negotiation successful
0901 20:01:51.762980 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":118,"threads_started":1}
I20250901 20:01:51.763548 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.757400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:39071 (local address 127.0.0.1:52366)
0901 20:01:51.758476 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:51.758541 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:51.758734 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:51.759530 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:51.759536 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:51.759561 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:51.759852 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.759863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.760952 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.760955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:51.761789 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.761799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.762711 (+ 912us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.762733 (+ 22us) client_negotiation.cc:769] Sending connection context
0901 20:01:51.763347 (+ 614us) client_negotiation.cc:240] Negotiation successful
0901 20:01:51.763421 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":37,"spinlock_wait_cycles":2944,"thread_start_us":133,"threads_started":1}
I20250901 20:01:51.763687 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.757696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.14.67.66:35979)
0901 20:01:51.759478 (+ 1782us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:51.759486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:51.759512 (+ 26us) server_negotiation.cc:407] Connection header received
0901 20:01:51.759606 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:51.759615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:51.759767 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:51.759914 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:51.760315 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.761361 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.762850 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.763363 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.763431 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:51.763481 (+ 50us) server_negotiation.cc:299] Negotiation successful
0901 20:01:51.763539 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1527,"thread_start_us":144,"threads_started":1}
I20250901 20:01:51.762748 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.757400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:35979 (local address 127.0.0.1:36200)
0901 20:01:51.758463 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:51.758542 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:51.758718 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:51.759928 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:51.759933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:51.759959 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:51.760196 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.760204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.761489 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.761495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:51.762339 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.762349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.762454 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.762471 (+ 17us) client_negotiation.cc:769] Sending connection context
0901 20:01:51.762547 (+ 76us) client_negotiation.cc:240] Negotiation successful
0901 20:01:51.762613 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"spinlock_wait_cycles":28544,"thread_start_us":145,"threads_started":1}
I20250901 20:01:51.765858 19427 tablet_service.cc:1468] Processing CreateTablet for tablet a38558df4df9409c97bb2354d9cf6836 (DEFAULT_TABLE table=test-table [id=7bb839a1af634a3e85280441f1954e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:51.765893 19697 tablet_service.cc:1468] Processing CreateTablet for tablet a38558df4df9409c97bb2354d9cf6836 (DEFAULT_TABLE table=test-table [id=7bb839a1af634a3e85280441f1954e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:51.765890 19562 tablet_service.cc:1468] Processing CreateTablet for tablet a38558df4df9409c97bb2354d9cf6836 (DEFAULT_TABLE table=test-table [id=7bb839a1af634a3e85280441f1954e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:51.766752 19427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a38558df4df9409c97bb2354d9cf6836. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:51.766801 19697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a38558df4df9409c97bb2354d9cf6836. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:51.766800 19562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a38558df4df9409c97bb2354d9cf6836. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:51.771955 19780 tablet_bootstrap.cc:492] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516: Bootstrap starting.
I20250901 20:01:51.773221 19781 tablet_bootstrap.cc:492] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94: Bootstrap starting.
I20250901 20:01:51.773628 19782 tablet_bootstrap.cc:492] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb: Bootstrap starting.
I20250901 20:01:51.774333 19780 tablet_bootstrap.cc:654] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:51.775053 19780 log.cc:826] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:51.775334 19781 tablet_bootstrap.cc:654] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:51.775609 19782 tablet_bootstrap.cc:654] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:51.776192 19781 log.cc:826] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:51.776310 19782 log.cc:826] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:51.776690 19780 tablet_bootstrap.cc:492] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516: No bootstrap required, opened a new log
I20250901 20:01:51.776890 19780 ts_tablet_manager.cc:1397] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250901 20:01:51.777978 19782 tablet_bootstrap.cc:492] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb: No bootstrap required, opened a new log
I20250901 20:01:51.777981 19781 tablet_bootstrap.cc:492] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94: No bootstrap required, opened a new log
I20250901 20:01:51.778158 19781 ts_tablet_manager.cc:1397] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250901 20:01:51.778158 19782 ts_tablet_manager.cc:1397] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250901 20:01:51.779680 19780 raft_consensus.cc:357] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b65d5640645f88e73f36d8d3e7516" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 39071 } } peers { permanent_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35979 } } peers { permanent_uuid: "d26c8cc255114587839f28b6ca4cef94" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40829 } }
I20250901 20:01:51.779868 19780 raft_consensus.cc:383] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:51.779943 19780 raft_consensus.cc:738] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100b65d5640645f88e73f36d8d3e7516, State: Initialized, Role: FOLLOWER
I20250901 20:01:51.780369 19780 consensus_queue.cc:260] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b65d5640645f88e73f36d8d3e7516" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 39071 } } peers { permanent_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35979 } } peers { permanent_uuid: "d26c8cc255114587839f28b6ca4cef94" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40829 } }
I20250901 20:01:51.781117 19780 ts_tablet_manager.cc:1428] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250901 20:01:51.781512 19493 heartbeater.cc:499] Master 127.14.67.126:44849 was elected leader, sending a full tablet report...
I20250901 20:01:51.781806 19782 raft_consensus.cc:357] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b65d5640645f88e73f36d8d3e7516" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 39071 } } peers { permanent_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35979 } } peers { permanent_uuid: "d26c8cc255114587839f28b6ca4cef94" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40829 } }
I20250901 20:01:51.781862 19781 raft_consensus.cc:357] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b65d5640645f88e73f36d8d3e7516" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 39071 } } peers { permanent_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35979 } } peers { permanent_uuid: "d26c8cc255114587839f28b6ca4cef94" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40829 } }
I20250901 20:01:51.782040 19781 raft_consensus.cc:383] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:51.782040 19782 raft_consensus.cc:383] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:51.782109 19782 raft_consensus.cc:738] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59aaf7d9556d446d857b3edbc7b4d9eb, State: Initialized, Role: FOLLOWER
I20250901 20:01:51.782153 19781 raft_consensus.cc:738] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26c8cc255114587839f28b6ca4cef94, State: Initialized, Role: FOLLOWER
I20250901 20:01:51.782605 19781 consensus_queue.cc:260] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b65d5640645f88e73f36d8d3e7516" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 39071 } } peers { permanent_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35979 } } peers { permanent_uuid: "d26c8cc255114587839f28b6ca4cef94" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40829 } }
I20250901 20:01:51.782696 19782 consensus_queue.cc:260] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b65d5640645f88e73f36d8d3e7516" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 39071 } } peers { permanent_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35979 } } peers { permanent_uuid: "d26c8cc255114587839f28b6ca4cef94" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40829 } }
I20250901 20:01:51.783492 19628 heartbeater.cc:499] Master 127.14.67.126:44849 was elected leader, sending a full tablet report...
I20250901 20:01:51.783548 19782 ts_tablet_manager.cc:1428] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250901 20:01:51.783546 19781 ts_tablet_manager.cc:1428] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250901 20:01:51.783913 19763 heartbeater.cc:499] Master 127.14.67.126:44849 was elected leader, sending a full tablet report...
I20250901 20:01:51.795650 19788 raft_consensus.cc:491] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:01:51.795809 19788 raft_consensus.cc:513] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b65d5640645f88e73f36d8d3e7516" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 39071 } } peers { permanent_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35979 } } peers { permanent_uuid: "d26c8cc255114587839f28b6ca4cef94" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40829 } }
I20250901 20:01:51.796818 19788 leader_election.cc:290] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100b65d5640645f88e73f36d8d3e7516 (127.14.67.65:39071), d26c8cc255114587839f28b6ca4cef94 (127.14.67.67:40829)
I20250901 20:01:51.799770 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.797058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:40829 (local address 127.14.67.66:49235)
0901 20:01:51.797178 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:51.797196 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:51.797293 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:51.797508 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:51.797511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:51.797529 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:51.797740 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.797747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.798711 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.798715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:51.799488 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.799496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.799582 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.799599 (+ 17us) client_negotiation.cc:769] Sending connection context
0901 20:01:51.799635 (+ 36us) client_negotiation.cc:240] Negotiation successful
0901 20:01:51.799679 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20250901 20:01:51.800365 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.797168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:49235 (local address 127.14.67.67:40829)
0901 20:01:51.797308 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:51.797311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:51.797323 (+ 12us) server_negotiation.cc:407] Connection header received
0901 20:01:51.797363 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:51.797366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:51.797405 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:51.797478 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:51.797872 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.798607 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.799609 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.800088 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.800127 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:51.800178 (+ 51us) server_negotiation.cc:299] Negotiation successful
0901 20:01:51.800241 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250901 20:01:51.800388 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.797060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:39071 (local address 127.14.67.66:44015)
0901 20:01:51.797665 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:51.797677 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:51.797760 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:51.797994 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:51.797998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:51.798018 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:01:51.798326 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.798334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.799240 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.799243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:51.799978 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:51.799988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.800112 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.800128 (+ 16us) client_negotiation.cc:769] Sending connection context
0901 20:01:51.800189 (+ 61us) client_negotiation.cc:240] Negotiation successful
0901 20:01:51.800255 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":296,"thread_start_us":127,"threads_started":1}
I20250901 20:01:51.800693 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.797114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:44015 (local address 127.14.67.65:39071)
0901 20:01:51.797245 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:51.797249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:51.797749 (+ 500us) server_negotiation.cc:407] Connection header received
0901 20:01:51.797843 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:51.797849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:51.797900 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:51.797960 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:01:51.798432 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.799100 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.800100 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.800415 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.800450 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:51.800504 (+ 54us) server_negotiation.cc:299] Negotiation successful
0901 20:01:51.800573 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250901 20:01:51.801019 19717 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38558df4df9409c97bb2354d9cf6836" candidate_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26c8cc255114587839f28b6ca4cef94" is_pre_election: true
I20250901 20:01:51.801204 19447 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38558df4df9409c97bb2354d9cf6836" candidate_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100b65d5640645f88e73f36d8d3e7516" is_pre_election: true
I20250901 20:01:51.801293 19717 raft_consensus.cc:2466] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59aaf7d9556d446d857b3edbc7b4d9eb in term 0.
I20250901 20:01:51.801429 19447 raft_consensus.cc:2466] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59aaf7d9556d446d857b3edbc7b4d9eb in term 0.
I20250901 20:01:51.801750 19518 leader_election.cc:304] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59aaf7d9556d446d857b3edbc7b4d9eb, d26c8cc255114587839f28b6ca4cef94; no voters:
I20250901 20:01:51.801957 19788 raft_consensus.cc:2802] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:01:51.802031 19788 raft_consensus.cc:491] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:01:51.802073 19788 raft_consensus.cc:3058] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:51.803138 19788 raft_consensus.cc:513] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b65d5640645f88e73f36d8d3e7516" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 39071 } } peers { permanent_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35979 } } peers { permanent_uuid: "d26c8cc255114587839f28b6ca4cef94" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40829 } }
I20250901 20:01:51.803514 19788 leader_election.cc:290] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [CANDIDATE]: Term 1 election: Requested vote from peers 100b65d5640645f88e73f36d8d3e7516 (127.14.67.65:39071), d26c8cc255114587839f28b6ca4cef94 (127.14.67.67:40829)
I20250901 20:01:51.803790 19447 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38558df4df9409c97bb2354d9cf6836" candidate_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100b65d5640645f88e73f36d8d3e7516"
I20250901 20:01:51.803874 19717 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38558df4df9409c97bb2354d9cf6836" candidate_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26c8cc255114587839f28b6ca4cef94"
I20250901 20:01:51.803910 19447 raft_consensus.cc:3058] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:51.803999 19717 raft_consensus.cc:3058] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:51.804915 19447 raft_consensus.cc:2466] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59aaf7d9556d446d857b3edbc7b4d9eb in term 1.
I20250901 20:01:51.805243 19516 leader_election.cc:304] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100b65d5640645f88e73f36d8d3e7516, 59aaf7d9556d446d857b3edbc7b4d9eb; no voters:
I20250901 20:01:51.805259 19717 raft_consensus.cc:2466] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59aaf7d9556d446d857b3edbc7b4d9eb in term 1.
I20250901 20:01:51.805410 19788 raft_consensus.cc:2802] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:51.805622 19788 raft_consensus.cc:695] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 1 LEADER]: Becoming Leader. State: Replica: 59aaf7d9556d446d857b3edbc7b4d9eb, State: Running, Role: LEADER
I20250901 20:01:51.806107 19788 consensus_queue.cc:237] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b65d5640645f88e73f36d8d3e7516" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 39071 } } peers { permanent_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35979 } } peers { permanent_uuid: "d26c8cc255114587839f28b6ca4cef94" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40829 } }
I20250901 20:01:51.808950 19301 catalog_manager.cc:5582] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 59aaf7d9556d446d857b3edbc7b4d9eb (127.14.67.66). New cstate: current_term: 1 leader_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b65d5640645f88e73f36d8d3e7516" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 39071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59aaf7d9556d446d857b3edbc7b4d9eb" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26c8cc255114587839f28b6ca4cef94" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40829 } health_report { overall_health: UNKNOWN } } }
I20250901 20:01:51.825631 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.822415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.14.67.66:35979)
0901 20:01:51.822572 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:51.822575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:51.822590 (+ 15us) server_negotiation.cc:407] Connection header received
0901 20:01:51.822724 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:51.822727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:51.822784 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:51.822861 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:01:51.823178 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.823668 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:51.824367 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:51.824515 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:51.824599 (+ 84us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:01:51.825036 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:01:51.825330 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:51.825379 (+ 49us) server_negotiation.cc:299] Negotiation successful
0901 20:01:51.825467 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250901 20:01:51.832652 19447 raft_consensus.cc:1273] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516 [term 1 FOLLOWER]: Refusing update from remote peer 59aaf7d9556d446d857b3edbc7b4d9eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:51.832652 19717 raft_consensus.cc:1273] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94 [term 1 FOLLOWER]: Refusing update from remote peer 59aaf7d9556d446d857b3edbc7b4d9eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:51.833572 19790 consensus_queue.cc:1035] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "100b65d5640645f88e73f36d8d3e7516" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:51.833806 19788 consensus_queue.cc:1035] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26c8cc255114587839f28b6ca4cef94" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 40829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:51.842063 19794 mvcc.cc:204] Tried to move back new op lower bound from 7195676310857678848 to 7195676310760214528. Current Snapshot: MvccSnapshot[applied={T|T < 7195676310857678848}]
I20250901 20:01:51.849637 19299 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48832:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:51.849779 19299 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48832:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:51.851908 19299 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 29dd0cc2c10740d38855a6a1c3618d4f: Sending DeleteTablet for 3 replicas of tablet a38558df4df9409c97bb2354d9cf6836
I20250901 20:01:51.852643 19427 tablet_service.cc:1515] Processing DeleteTablet for tablet a38558df4df9409c97bb2354d9cf6836 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:51 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:52366
I20250901 20:01:51.852638 19562 tablet_service.cc:1515] Processing DeleteTablet for tablet a38558df4df9409c97bb2354d9cf6836 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:51 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:36200
I20250901 20:01:51.853093 19804 tablet_replica.cc:331] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516: stopping tablet replica
I20250901 20:01:51.853116 19803 tablet_replica.cc:331] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb: stopping tablet replica
I20250901 20:01:51.853266 19697 tablet_service.cc:1515] Processing DeleteTablet for tablet a38558df4df9409c97bb2354d9cf6836 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:51 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:35958
I20250901 20:01:51.853359 19803 raft_consensus.cc:2241] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:01:51.853394 19804 raft_consensus.cc:2241] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:51.853701 19804 raft_consensus.cc:2270] T a38558df4df9409c97bb2354d9cf6836 P 100b65d5640645f88e73f36d8d3e7516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:51.853807 19803 raft_consensus.cc:2270] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:51.854269 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 19365
I20250901 20:01:51.855707 19803 ts_tablet_manager.cc:1905] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:51.856607 19805 tablet_replica.cc:331] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94: stopping tablet replica
I20250901 20:01:51.856949 19805 raft_consensus.cc:2241] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:51.857301 19805 raft_consensus.cc:2270] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:51.858649 19803 ts_tablet_manager.cc:1918] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:01:51.858709 19805 ts_tablet_manager.cc:1905] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:51.858793 19803 log.cc:1199] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/a38558df4df9409c97bb2354d9cf6836
I20250901 20:01:51.859206 19803 ts_tablet_manager.cc:1939] T a38558df4df9409c97bb2354d9cf6836 P 59aaf7d9556d446d857b3edbc7b4d9eb: Deleting consensus metadata
I20250901 20:01:51.860394 19285 catalog_manager.cc:4928] TS 59aaf7d9556d446d857b3edbc7b4d9eb (127.14.67.66:35979): tablet a38558df4df9409c97bb2354d9cf6836 (table test-table [id=7bb839a1af634a3e85280441f1954e5e]) successfully deleted
I20250901 20:01:51.862247 19805 ts_tablet_manager.cc:1918] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:01:51.862397 19805 log.cc:1199] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/a38558df4df9409c97bb2354d9cf6836
W20250901 20:01:51.862354 19286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.67.65:39071 (error 108)
I20250901 20:01:51.862826 19805 ts_tablet_manager.cc:1939] T a38558df4df9409c97bb2354d9cf6836 P d26c8cc255114587839f28b6ca4cef94: Deleting consensus metadata
I20250901 20:01:51.863385 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:51.863075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:39071 (local address 127.0.0.1:52374)
0901 20:01:51.863226 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:51.863301 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.65:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20250901 20:01:51.863423 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 19500
I20250901 20:01:51.864395 19288 catalog_manager.cc:4928] TS d26c8cc255114587839f28b6ca4cef94 (127.14.67.67:40829): tablet a38558df4df9409c97bb2354d9cf6836 (table test-table [id=7bb839a1af634a3e85280441f1954e5e]) successfully deleted
W20250901 20:01:51.865370 19286 catalog_manager.cc:4671] TS 100b65d5640645f88e73f36d8d3e7516 (127.14.67.65:39071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a38558df4df9409c97bb2354d9cf6836: Network error: Client connection negotiation failed: client connection to 127.14.67.65:39071: connect: Connection refused (error 111)
I20250901 20:01:51.869587 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 19635
I20250901 20:01:51.875084 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 19270
2025-09-01T20:01:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3200 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-09-01T20:01:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:01:51Z Disabled control of system clock
I20250901 20:01:51.905146 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:40875
--webserver_interface=127.14.67.126
--webserver_port=0
--builtin_ntp_servers=127.14.67.84:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:40875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250901 20:01:52.004820 19810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:52.005030 19810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:52.005075 19810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:52.008077 19810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:01:52.008128 19810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:52.008145 19810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:52.008163 19810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:01:52.008180 19810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:01:52.012159 19810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:40875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:40875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:52.013113 19810 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:52.014125 19810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:52.019897 19816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:52.019909 19818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:52.020090 19810 server_base.cc:1047] running on GCE node
W20250901 20:01:52.019934 19815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:52.020691 19810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:52.021515 19810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:52.022665 19810 hybrid_clock.cc:648] HybridClock initialized: now 1756756912022645 us; error 35 us; skew 500 ppm
I20250901 20:01:52.024750 19810 webserver.cc:480] Webserver started at http://127.14.67.126:41045/ using document root <none> and password file <none>
I20250901 20:01:52.025277 19810 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:52.025348 19810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:52.025557 19810 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:52.027261 19810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "9b13bda4c1174d80a2296a1fdc124ef8"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "54ad6fdc1571c986346e4ee1231e2462"
server_key_iv: "3fda2f3552aaa7506cdf68c955e27f91"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.027721 19810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "9b13bda4c1174d80a2296a1fdc124ef8"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "54ad6fdc1571c986346e4ee1231e2462"
server_key_iv: "3fda2f3552aaa7506cdf68c955e27f91"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.031009 19810 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250901 20:01:52.033210 19824 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:52.034201 19810 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:01:52.034312 19810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "9b13bda4c1174d80a2296a1fdc124ef8"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "54ad6fdc1571c986346e4ee1231e2462"
server_key_iv: "3fda2f3552aaa7506cdf68c955e27f91"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.034413 19810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:52.039021 19810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:52.039570 19810 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:52.039729 19810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:52.046451 19876 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:40875 every 8 connection(s)
I20250901 20:01:52.046449 19810 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:40875
I20250901 20:01:52.047446 19810 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:01:52.049511 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 19810
I20250901 20:01:52.049638 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:01:52.049890 14605 external_mini_cluster.cc:1468] Setting key 7e8745f63f5be3ac1e4464cb09340e48
I20250901 20:01:52.050058 19877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:52.057294 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.051620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.14.67.126:40875)
0901 20:01:52.052534 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:52.052544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:52.052586 (+ 42us) server_negotiation.cc:407] Connection header received
0901 20:01:52.053841 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:52.053868 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:52.054415 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:52.054816 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:52.055994 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:52.056000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:52.056011 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:52.056023 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:52.056071 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:52.056084 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:52.056245 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:52.056405 (+ 160us) server_negotiation.cc:299] Negotiation successful
0901 20:01:52.056641 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":157,"threads_started":1}
I20250901 20:01:52.057461 19877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8: Bootstrap starting.
I20250901 20:01:52.059691 19877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:52.060317 19877 log.cc:826] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:52.061931 19877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8: No bootstrap required, opened a new log
I20250901 20:01:52.064184 19877 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b13bda4c1174d80a2296a1fdc124ef8" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 40875 } }
I20250901 20:01:52.064358 19877 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:52.064399 19877 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b13bda4c1174d80a2296a1fdc124ef8, State: Initialized, Role: FOLLOWER
I20250901 20:01:52.064803 19877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b13bda4c1174d80a2296a1fdc124ef8" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 40875 } }
I20250901 20:01:52.064925 19877 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:01:52.064963 19877 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:01:52.065057 19877 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:52.065970 19877 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b13bda4c1174d80a2296a1fdc124ef8" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 40875 } }
I20250901 20:01:52.066265 19877 leader_election.cc:304] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b13bda4c1174d80a2296a1fdc124ef8; no voters:
I20250901 20:01:52.066548 19877 leader_election.cc:290] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:01:52.066699 19882 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:52.066895 19882 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [term 1 LEADER]: Becoming Leader. State: Replica: 9b13bda4c1174d80a2296a1fdc124ef8, State: Running, Role: LEADER
I20250901 20:01:52.067201 19882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b13bda4c1174d80a2296a1fdc124ef8" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 40875 } }
I20250901 20:01:52.067533 19877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:01:52.068783 19883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b13bda4c1174d80a2296a1fdc124ef8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b13bda4c1174d80a2296a1fdc124ef8" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 40875 } } }
I20250901 20:01:52.068881 19883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:52.069119 19884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b13bda4c1174d80a2296a1fdc124ef8. Latest consensus state: current_term: 1 leader_uuid: "9b13bda4c1174d80a2296a1fdc124ef8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b13bda4c1174d80a2296a1fdc124ef8" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 40875 } } }
I20250901 20:01:52.069257 19891 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:01:52.069257 19884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:52.072769 19891 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:01:52.078881 19891 catalog_manager.cc:1349] Generated new cluster ID: 543c700b22214c8bbdcab052f1d73990
I20250901 20:01:52.078966 19891 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:01:52.094575 19891 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:01:52.094902 19891 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:01:52.101786 19891 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8: Generated new TSK 0
I20250901 20:01:52.102541 19891 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 20:01:52.111630 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:40875
--builtin_ntp_servers=127.14.67.84:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250901 20:01:52.222807 19901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:52.223002 19901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:52.223053 19901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:52.226114 19901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:52.226164 19901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:52.226341 19901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:01:52.230022 19901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:40875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:52.231088 19901 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:52.232227 19901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:52.238488 19907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:52.238616 19906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:52.238616 19909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:52.238961 19901 server_base.cc:1047] running on GCE node
I20250901 20:01:52.239300 19901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:52.239792 19901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:52.240969 19901 hybrid_clock.cc:648] HybridClock initialized: now 1756756912240954 us; error 34 us; skew 500 ppm
I20250901 20:01:52.242800 19901 webserver.cc:480] Webserver started at http://127.14.67.65:37705/ using document root <none> and password file <none>
I20250901 20:01:52.243345 19901 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:52.243407 19901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:52.243583 19901 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:52.245246 19901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "9bcce3deb1f948a9b22a1219ed7b7032"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "ebdb9009283d0bb22ef5200751aef0da"
server_key_iv: "51bbf3501c2bd4d885102cea73e6b1df"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.245712 19901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "9bcce3deb1f948a9b22a1219ed7b7032"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "ebdb9009283d0bb22ef5200751aef0da"
server_key_iv: "51bbf3501c2bd4d885102cea73e6b1df"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.249166 19901 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250901 20:01:52.251596 19915 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:52.252645 19901 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250901 20:01:52.252770 19901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "9bcce3deb1f948a9b22a1219ed7b7032"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "ebdb9009283d0bb22ef5200751aef0da"
server_key_iv: "51bbf3501c2bd4d885102cea73e6b1df"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.252866 19901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:52.258373 19901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:52.259063 19901 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:52.259225 19901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:52.259759 19901 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:52.260663 19901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:52.260712 19901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:52.260847 19901 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:52.260900 19901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:52.269677 19901 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:42389
I20250901 20:01:52.269824 20028 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:42389 every 8 connection(s)
I20250901 20:01:52.270617 19901 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:01:52.275707 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.272372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:54735 (local address 127.14.67.126:40875)
0901 20:01:52.272527 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:52.272532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:52.273455 (+ 923us) server_negotiation.cc:407] Connection header received
0901 20:01:52.274181 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:52.274184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:52.274221 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:52.274295 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:52.275354 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:52.275357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:52.275359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:52.275368 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:52.275385 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:52.275392 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:52.275462 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:52.275568 (+ 106us) server_negotiation.cc:299] Negotiation successful
0901 20:01:52.275604 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250901 20:01:52.276367 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.272680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:40875 (local address 127.14.67.65:54735)
0901 20:01:52.273308 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:52.273357 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:52.274005 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:52.274439 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:52.274446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:52.274564 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:52.275141 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:01:52.275143 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:52.275172 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:01:52.275222 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:52.275459 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:52.275463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:52.275467 (+ 4us) client_negotiation.cc:769] Sending connection context
0901 20:01:52.275597 (+ 130us) client_negotiation.cc:240] Negotiation successful
0901 20:01:52.275749 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":102,"threads_started":1}
I20250901 20:01:52.277385 20029 heartbeater.cc:344] Connected to a master server at 127.14.67.126:40875
I20250901 20:01:52.277593 20029 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:52.277976 20029 heartbeater.cc:507] Master 127.14.67.126:40875 requested a full tablet report, sending...
I20250901 20:01:52.278321 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 19901
I20250901 20:01:52.278438 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:01:52.278761 14605 external_mini_cluster.cc:1468] Setting key c1f1ba230217219804df0a2d7b84daf0
I20250901 20:01:52.279479 19841 ts_manager.cc:194] Registered new tserver with Master: 9bcce3deb1f948a9b22a1219ed7b7032 (127.14.67.65:42389)
I20250901 20:01:52.280632 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:40875
--builtin_ntp_servers=127.14.67.84:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20250901 20:01:52.384248 20032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:52.384467 20032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:52.384553 20032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:52.387601 20032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:52.387651 20032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:52.387765 20032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:01:52.391259 20032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:40875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:52.392289 20032 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:52.393355 20032 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:52.399582 20037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:52.399596 20040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:52.399596 20038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:52.399652 20032 server_base.cc:1047] running on GCE node
I20250901 20:01:52.400143 20032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:52.400633 20032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:52.401806 20032 hybrid_clock.cc:648] HybridClock initialized: now 1756756912401787 us; error 35 us; skew 500 ppm
I20250901 20:01:52.403452 20032 webserver.cc:480] Webserver started at http://127.14.67.66:33623/ using document root <none> and password file <none>
I20250901 20:01:52.403939 20032 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:52.403987 20032 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:52.404130 20032 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:52.405673 20032 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "27af2e6d2a6b402fac92be70909c8c52"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "3e3ece5d538974c19b0cf56fcad89ada"
server_key_iv: "ecb220d389bae684d57bb9de38fee79d"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.406072 20032 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "27af2e6d2a6b402fac92be70909c8c52"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "3e3ece5d538974c19b0cf56fcad89ada"
server_key_iv: "ecb220d389bae684d57bb9de38fee79d"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.409446 20032 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:01:52.411887 20046 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:52.413009 20032 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:01:52.413178 20032 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "27af2e6d2a6b402fac92be70909c8c52"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "3e3ece5d538974c19b0cf56fcad89ada"
server_key_iv: "ecb220d389bae684d57bb9de38fee79d"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.413295 20032 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:52.419319 20032 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:52.419963 20032 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:52.420130 20032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:52.420675 20032 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:52.421852 20032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:52.421913 20032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:52.421957 20032 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:52.421979 20032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:52.431476 20032 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:33711
I20250901 20:01:52.431509 20159 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:33711 every 8 connection(s)
I20250901 20:01:52.432765 20032 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:01:52.435932 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 20032
I20250901 20:01:52.436069 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:01:52.436344 14605 external_mini_cluster.cc:1468] Setting key 1414e47779a35eebb126df45e0f2b0f0
I20250901 20:01:52.438234 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:40875
--builtin_ntp_servers=127.14.67.84:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20250901 20:01:52.439354 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.434643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:43055 (local address 127.14.67.126:40875)
0901 20:01:52.434799 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:52.434804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:52.435731 (+ 927us) server_negotiation.cc:407] Connection header received
0901 20:01:52.436720 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:52.436725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:52.436774 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:52.436858 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:52.438900 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:52.438902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:52.438904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:52.438915 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:52.438933 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:52.438940 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:52.439017 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:52.439178 (+ 161us) server_negotiation.cc:299] Negotiation successful
0901 20:01:52.439222 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250901 20:01:52.440131 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.434935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:40875 (local address 127.14.67.66:43055)
0901 20:01:52.435566 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:52.435610 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:52.436505 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:52.437014 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:52.437024 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:52.437114 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:52.438631 (+ 1517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:01:52.438632 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:52.438662 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:01:52.438738 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:52.439017 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:52.439028 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:52.439036 (+ 8us) client_negotiation.cc:769] Sending connection context
0901 20:01:52.439210 (+ 174us) client_negotiation.cc:240] Negotiation successful
0901 20:01:52.439508 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":123,"threads_started":1}
I20250901 20:01:52.441104 20160 heartbeater.cc:344] Connected to a master server at 127.14.67.126:40875
I20250901 20:01:52.441330 20160 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:52.441730 20160 heartbeater.cc:507] Master 127.14.67.126:40875 requested a full tablet report, sending...
I20250901 20:01:52.442696 19841 ts_manager.cc:194] Registered new tserver with Master: 27af2e6d2a6b402fac92be70909c8c52 (127.14.67.66:33711)
W20250901 20:01:52.539917 20163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:52.540205 20163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:52.540258 20163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:52.543979 20163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:52.544039 20163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:52.544118 20163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:01:52.548703 20163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:40875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:52.550216 20163 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:52.551760 20163 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:52.558837 20168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:52.558916 20163 server_base.cc:1047] running on GCE node
W20250901 20:01:52.558837 20169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:52.558832 20171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:52.559505 20163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:52.560035 20163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:52.561183 20163 hybrid_clock.cc:648] HybridClock initialized: now 1756756912561168 us; error 40 us; skew 500 ppm
I20250901 20:01:52.562954 20163 webserver.cc:480] Webserver started at http://127.14.67.67:36797/ using document root <none> and password file <none>
I20250901 20:01:52.563504 20163 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:52.563558 20163 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:52.563743 20163 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:52.565402 20163 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "5fd626b8a1154352bd0be826957b4124"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "a64430b946502dcd64d7e38a859edd05"
server_key_iv: "93a7bb169fc4c9c52e7b1eb56ce7614c"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.565852 20163 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "5fd626b8a1154352bd0be826957b4124"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "a64430b946502dcd64d7e38a859edd05"
server_key_iv: "93a7bb169fc4c9c52e7b1eb56ce7614c"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.569502 20163 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:01:52.572039 20177 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:52.573076 20163 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:01:52.573180 20163 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "5fd626b8a1154352bd0be826957b4124"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "a64430b946502dcd64d7e38a859edd05"
server_key_iv: "93a7bb169fc4c9c52e7b1eb56ce7614c"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.573297 20163 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:52.578217 20163 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:52.578812 20163 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:52.578984 20163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:52.579484 20163 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:01:52.580349 20163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:01:52.580399 20163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:52.580435 20163 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:01:52.580449 20163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:52.588543 20163 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:42667
I20250901 20:01:52.588572 20290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:42667 every 8 connection(s)
I20250901 20:01:52.589432 20163 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:01:52.593607 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 20163
I20250901 20:01:52.593710 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:01:52.593992 14605 external_mini_cluster.cc:1468] Setting key 8c6e1a936c7a07e74efdc9a0afb4f72f
I20250901 20:01:52.594681 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.591059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:60507 (local address 127.14.67.126:40875)
0901 20:01:52.591201 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:52.591206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:52.592065 (+ 859us) server_negotiation.cc:407] Connection header received
0901 20:01:52.592830 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:52.592833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:52.592936 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:52.593012 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:52.594191 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:52.594194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:52.594197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:52.594207 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:52.594226 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:52.594233 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:52.594328 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:52.594469 (+ 141us) server_negotiation.cc:299] Negotiation successful
0901 20:01:52.594550 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250901 20:01:52.595387 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.591348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:40875 (local address 127.14.67.67:60507)
0901 20:01:52.591934 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:52.591968 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:52.592629 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:52.593179 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:52.593186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:52.593269 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:52.593933 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:01:52.593937 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:52.593975 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:01:52.594041 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:52.594320 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:52.594327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:52.594331 (+ 4us) client_negotiation.cc:769] Sending connection context
0901 20:01:52.594495 (+ 164us) client_negotiation.cc:240] Negotiation successful
0901 20:01:52.594741 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
I20250901 20:01:52.596572 20291 heartbeater.cc:344] Connected to a master server at 127.14.67.126:40875
I20250901 20:01:52.596794 20291 heartbeater.cc:461] Registering TS with master...
I20250901 20:01:52.597134 20291 heartbeater.cc:507] Master 127.14.67.126:40875 requested a full tablet report, sending...
I20250901 20:01:52.598098 19840 ts_manager.cc:194] Registered new tserver with Master: 5fd626b8a1154352bd0be826957b4124 (127.14.67.67:42667)
I20250901 20:01:52.607815 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250901 20:01:52.611617 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.609936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.14.67.126:40875)
0901 20:01:52.610097 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:52.610102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:52.610234 (+ 132us) server_negotiation.cc:407] Connection header received
0901 20:01:52.610389 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:52.610393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:52.610441 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:52.610544 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:52.611237 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:52.611241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:52.611244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:52.611255 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:52.611275 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:52.611285 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:52.611368 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:52.611442 (+ 74us) server_negotiation.cc:299] Negotiation successful
0901 20:01:52.611482 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20250901 20:01:52.614369 19840 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 20:01:52.616735 19840 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:01:52.627848 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.624869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:42667 (local address 127.0.0.1:38560)
0901 20:01:52.625319 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:52.625332 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:52.625406 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:52.626157 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:52.626159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:52.626170 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:52.627258 (+ 1088us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:01:52.627261 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:52.627274 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:01:52.627300 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:52.627584 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:52.627587 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:52.627590 (+ 3us) client_negotiation.cc:769] Sending connection context
0901 20:01:52.627657 (+ 67us) client_negotiation.cc:240] Negotiation successful
0901 20:01:52.627693 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":86,"threads_started":1}
I20250901 20:01:52.627880 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.624528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:42389 (local address 127.0.0.1:46514)
0901 20:01:52.624862 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:52.624902 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:52.624996 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:52.626096 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:52.626103 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:52.626171 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:52.627258 (+ 1087us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:01:52.627261 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:52.627274 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:01:52.627300 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:52.627640 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:52.627645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:52.627647 (+ 2us) client_negotiation.cc:769] Sending connection context
0901 20:01:52.627694 (+ 47us) client_negotiation.cc:240] Negotiation successful
0901 20:01:52.627718 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20250901 20:01:52.628180 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.625276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.14.67.67:42667)
0901 20:01:52.625760 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:52.625768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:52.625793 (+ 25us) server_negotiation.cc:407] Connection header received
0901 20:01:52.625848 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:52.625855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:52.626000 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:52.626174 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:52.627421 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:52.627429 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:52.627440 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:52.627455 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:52.627497 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:52.627508 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:52.627620 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:52.627662 (+ 42us) server_negotiation.cc:299] Negotiation successful
0901 20:01:52.627724 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":143,"threads_started":1}
I20250901 20:01:52.627880 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.625252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:33711 (local address 127.0.0.1:38518)
0901 20:01:52.625475 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:52.625489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:52.625885 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:52.626546 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:52.626550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:52.626564 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:52.627084 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:01:52.627086 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:52.627118 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:01:52.627179 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:52.627476 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:52.627483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:52.627489 (+ 6us) client_negotiation.cc:769] Sending connection context
0901 20:01:52.627620 (+ 131us) client_negotiation.cc:240] Negotiation successful
0901 20:01:52.627684 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20250901 20:01:52.628947 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.624713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.14.67.65:42389)
0901 20:01:52.625659 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:52.625666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:52.625689 (+ 23us) server_negotiation.cc:407] Connection header received
0901 20:01:52.625743 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:52.625750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:52.625920 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:52.626341 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:52.627445 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:52.627452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:52.627464 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:52.627475 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:52.627526 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:52.627539 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:52.628099 (+ 560us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:52.628357 (+ 258us) server_negotiation.cc:299] Negotiation successful
0901 20:01:52.628417 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":199,"threads_started":1}
I20250901 20:01:52.629382 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.625636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.14.67.66:33711)
0901 20:01:52.626131 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:52.626138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:52.626157 (+ 19us) server_negotiation.cc:407] Connection header received
0901 20:01:52.626205 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:52.626210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:52.626354 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:52.626775 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:52.627302 (+ 527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:52.627307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:52.627318 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:52.627328 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:52.627378 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:52.627392 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:52.627507 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:52.628950 (+ 1443us) server_negotiation.cc:299] Negotiation successful
0901 20:01:52.629018 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":115,"threads_started":1}
I20250901 20:01:52.630419 20225 tablet_service.cc:1468] Processing CreateTablet for tablet 0fcab025afb047d09352e336af6a9a1c (DEFAULT_TABLE table=test-table [id=d17e0a3e8b0c44ddbe73f7a5a3ca282f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:52.631402 20225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fcab025afb047d09352e336af6a9a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:52.631227 20093 tablet_service.cc:1468] Processing CreateTablet for tablet 0fcab025afb047d09352e336af6a9a1c (DEFAULT_TABLE table=test-table [id=d17e0a3e8b0c44ddbe73f7a5a3ca282f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:52.632040 20093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fcab025afb047d09352e336af6a9a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:52.632572 19963 tablet_service.cc:1468] Processing CreateTablet for tablet 0fcab025afb047d09352e336af6a9a1c (DEFAULT_TABLE table=test-table [id=d17e0a3e8b0c44ddbe73f7a5a3ca282f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:01:52.633366 19963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fcab025afb047d09352e336af6a9a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:52.636365 20308 tablet_bootstrap.cc:492] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124: Bootstrap starting.
I20250901 20:01:52.637770 20309 tablet_bootstrap.cc:492] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52: Bootstrap starting.
I20250901 20:01:52.639091 20310 tablet_bootstrap.cc:492] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032: Bootstrap starting.
I20250901 20:01:52.639698 20308 tablet_bootstrap.cc:654] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:52.639698 20309 tablet_bootstrap.cc:654] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:52.640544 20309 log.cc:826] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:52.640544 20308 log.cc:826] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:52.640741 20310 tablet_bootstrap.cc:654] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:52.641486 20310 log.cc:826] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:52.642153 20308 tablet_bootstrap.cc:492] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124: No bootstrap required, opened a new log
I20250901 20:01:52.642318 20308 ts_tablet_manager.cc:1397] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250901 20:01:52.642946 20310 tablet_bootstrap.cc:492] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032: No bootstrap required, opened a new log
I20250901 20:01:52.643121 20310 ts_tablet_manager.cc:1397] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250901 20:01:52.643878 20309 tablet_bootstrap.cc:492] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52: No bootstrap required, opened a new log
I20250901 20:01:52.644054 20309 ts_tablet_manager.cc:1397] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250901 20:01:52.644816 20308 raft_consensus.cc:357] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 42389 } } peers { permanent_uuid: "27af2e6d2a6b402fac92be70909c8c52" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 33711 } } peers { permanent_uuid: "5fd626b8a1154352bd0be826957b4124" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42667 } }
I20250901 20:01:52.644989 20308 raft_consensus.cc:383] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:52.645042 20308 raft_consensus.cc:738] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd626b8a1154352bd0be826957b4124, State: Initialized, Role: FOLLOWER
I20250901 20:01:52.645432 20308 consensus_queue.cc:260] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 42389 } } peers { permanent_uuid: "27af2e6d2a6b402fac92be70909c8c52" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 33711 } } peers { permanent_uuid: "5fd626b8a1154352bd0be826957b4124" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42667 } }
I20250901 20:01:52.645942 20310 raft_consensus.cc:357] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 42389 } } peers { permanent_uuid: "27af2e6d2a6b402fac92be70909c8c52" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 33711 } } peers { permanent_uuid: "5fd626b8a1154352bd0be826957b4124" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42667 } }
I20250901 20:01:52.646118 20310 raft_consensus.cc:383] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:52.646162 20310 raft_consensus.cc:738] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcce3deb1f948a9b22a1219ed7b7032, State: Initialized, Role: FOLLOWER
I20250901 20:01:52.646278 20308 ts_tablet_manager.cc:1428] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250901 20:01:52.646586 20310 consensus_queue.cc:260] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 42389 } } peers { permanent_uuid: "27af2e6d2a6b402fac92be70909c8c52" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 33711 } } peers { permanent_uuid: "5fd626b8a1154352bd0be826957b4124" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42667 } }
I20250901 20:01:52.646654 20291 heartbeater.cc:499] Master 127.14.67.126:40875 was elected leader, sending a full tablet report...
I20250901 20:01:52.646914 20309 raft_consensus.cc:357] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 42389 } } peers { permanent_uuid: "27af2e6d2a6b402fac92be70909c8c52" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 33711 } } peers { permanent_uuid: "5fd626b8a1154352bd0be826957b4124" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42667 } }
I20250901 20:01:52.647083 20309 raft_consensus.cc:383] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:52.647133 20309 raft_consensus.cc:738] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27af2e6d2a6b402fac92be70909c8c52, State: Initialized, Role: FOLLOWER
I20250901 20:01:52.647177 20029 heartbeater.cc:499] Master 127.14.67.126:40875 was elected leader, sending a full tablet report...
I20250901 20:01:52.647365 20310 ts_tablet_manager.cc:1428] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250901 20:01:52.647531 20309 consensus_queue.cc:260] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 42389 } } peers { permanent_uuid: "27af2e6d2a6b402fac92be70909c8c52" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 33711 } } peers { permanent_uuid: "5fd626b8a1154352bd0be826957b4124" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42667 } }
I20250901 20:01:52.648118 20160 heartbeater.cc:499] Master 127.14.67.126:40875 was elected leader, sending a full tablet report...
I20250901 20:01:52.648293 20309 ts_tablet_manager.cc:1428] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
W20250901 20:01:52.683835 20161 tablet.cc:2378] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 20:01:52.702040 20315 raft_consensus.cc:491] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:01:52.702231 20315 raft_consensus.cc:513] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 42389 } } peers { permanent_uuid: "27af2e6d2a6b402fac92be70909c8c52" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 33711 } } peers { permanent_uuid: "5fd626b8a1154352bd0be826957b4124" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42667 } }
I20250901 20:01:52.703280 20315 leader_election.cc:290] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27af2e6d2a6b402fac92be70909c8c52 (127.14.67.66:33711), 5fd626b8a1154352bd0be826957b4124 (127.14.67.67:42667)
I20250901 20:01:52.705154 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.703654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:57403 (local address 127.14.67.67:42667)
0901 20:01:52.703799 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:52.703803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:52.703819 (+ 16us) server_negotiation.cc:407] Connection header received
0901 20:01:52.703867 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:52.703870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:52.703910 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:52.703990 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:52.704752 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:52.704755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:52.704757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:52.704767 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:52.704785 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:52.704792 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:52.704892 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:52.704981 (+ 89us) server_negotiation.cc:299] Negotiation successful
0901 20:01:52.705023 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20250901 20:01:52.705179 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.703536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:42667 (local address 127.14.67.65:57403)
0901 20:01:52.703670 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:52.703687 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:52.703767 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:52.704043 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:52.704046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:52.704058 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:52.704590 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:01:52.704592 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:52.704602 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:01:52.704626 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:52.704874 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:52.704879 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:52.704884 (+ 5us) client_negotiation.cc:769] Sending connection context
0901 20:01:52.704976 (+ 92us) client_negotiation.cc:240] Negotiation successful
0901 20:01:52.705021 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20250901 20:01:52.705857 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.703760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:58565 (local address 127.14.67.66:33711)
0901 20:01:52.703895 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:52.703899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:52.704063 (+ 164us) server_negotiation.cc:407] Connection header received
0901 20:01:52.704141 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:52.704143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:52.704176 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:52.704253 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:52.705413 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:52.705416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:52.705418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:52.705431 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:52.705451 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:52.705460 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:52.705518 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:52.705701 (+ 183us) server_negotiation.cc:299] Negotiation successful
0901 20:01:52.705740 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250901 20:01:52.705813 20245 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fcab025afb047d09352e336af6a9a1c" candidate_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd626b8a1154352bd0be826957b4124" is_pre_election: true
I20250901 20:01:52.706079 20245 raft_consensus.cc:2466] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bcce3deb1f948a9b22a1219ed7b7032 in term 0.
I20250901 20:01:52.706207 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.703536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:33711 (local address 127.14.67.65:58565)
0901 20:01:52.703969 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:52.703985 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:52.704059 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:52.704275 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:52.704278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:52.704289 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:01:52.705229 (+ 940us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:01:52.705231 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:01:52.705243 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:01:52.705265 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:01:52.705597 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:01:52.705601 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:01:52.705606 (+ 5us) client_negotiation.cc:769] Sending connection context
0901 20:01:52.706008 (+ 402us) client_negotiation.cc:240] Negotiation successful
0901 20:01:52.706044 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":51,"thread_start_us":138,"threads_started":1}
I20250901 20:01:52.706560 19918 leader_election.cc:304] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fd626b8a1154352bd0be826957b4124, 9bcce3deb1f948a9b22a1219ed7b7032; no voters:
I20250901 20:01:52.706785 20315 raft_consensus.cc:2802] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:01:52.706810 20114 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fcab025afb047d09352e336af6a9a1c" candidate_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27af2e6d2a6b402fac92be70909c8c52" is_pre_election: true
I20250901 20:01:52.706876 20315 raft_consensus.cc:491] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:01:52.706933 20315 raft_consensus.cc:3058] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:52.707041 20114 raft_consensus.cc:2466] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bcce3deb1f948a9b22a1219ed7b7032 in term 0.
I20250901 20:01:52.708025 20315 raft_consensus.cc:513] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 42389 } } peers { permanent_uuid: "27af2e6d2a6b402fac92be70909c8c52" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 33711 } } peers { permanent_uuid: "5fd626b8a1154352bd0be826957b4124" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42667 } }
I20250901 20:01:52.708339 20315 leader_election.cc:290] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [CANDIDATE]: Term 1 election: Requested vote from peers 27af2e6d2a6b402fac92be70909c8c52 (127.14.67.66:33711), 5fd626b8a1154352bd0be826957b4124 (127.14.67.67:42667)
I20250901 20:01:52.708612 20114 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fcab025afb047d09352e336af6a9a1c" candidate_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27af2e6d2a6b402fac92be70909c8c52"
I20250901 20:01:52.708726 20114 raft_consensus.cc:3058] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:52.709662 20245 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fcab025afb047d09352e336af6a9a1c" candidate_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd626b8a1154352bd0be826957b4124"
I20250901 20:01:52.709769 20245 raft_consensus.cc:3058] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:52.710031 20114 raft_consensus.cc:2466] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bcce3deb1f948a9b22a1219ed7b7032 in term 1.
I20250901 20:01:52.710330 19916 leader_election.cc:304] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27af2e6d2a6b402fac92be70909c8c52, 9bcce3deb1f948a9b22a1219ed7b7032; no voters:
I20250901 20:01:52.710568 20315 raft_consensus.cc:2802] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:52.710791 20315 raft_consensus.cc:695] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 1 LEADER]: Becoming Leader. State: Replica: 9bcce3deb1f948a9b22a1219ed7b7032, State: Running, Role: LEADER
I20250901 20:01:52.710927 20245 raft_consensus.cc:2466] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bcce3deb1f948a9b22a1219ed7b7032 in term 1.
I20250901 20:01:52.711092 20315 consensus_queue.cc:237] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 42389 } } peers { permanent_uuid: "27af2e6d2a6b402fac92be70909c8c52" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 33711 } } peers { permanent_uuid: "5fd626b8a1154352bd0be826957b4124" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42667 } }
I20250901 20:01:52.714015 19840 catalog_manager.cc:5582] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bcce3deb1f948a9b22a1219ed7b7032 (127.14.67.65). New cstate: current_term: 1 leader_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bcce3deb1f948a9b22a1219ed7b7032" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 42389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27af2e6d2a6b402fac92be70909c8c52" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 33711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fd626b8a1154352bd0be826957b4124" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42667 } health_report { overall_health: UNKNOWN } } }
I20250901 20:01:52.730261 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.728986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.14.67.65:42389)
0901 20:01:52.729105 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:52.729108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:52.729122 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:01:52.729191 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:52.729193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:52.729245 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:52.729317 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:52.729962 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:52.729965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:52.729967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:52.729977 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:52.729994 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:52.730001 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:52.730073 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:52.730130 (+ 57us) server_negotiation.cc:299] Negotiation successful
0901 20:01:52.730161 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20250901 20:01:52.735208 20245 raft_consensus.cc:1273] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124 [term 1 FOLLOWER]: Refusing update from remote peer 9bcce3deb1f948a9b22a1219ed7b7032: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:52.735208 20114 raft_consensus.cc:1273] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52 [term 1 FOLLOWER]: Refusing update from remote peer 9bcce3deb1f948a9b22a1219ed7b7032: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:01:52.735783 20318 consensus_queue.cc:1035] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fd626b8a1154352bd0be826957b4124" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:52.735947 20315 consensus_queue.cc:1035] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27af2e6d2a6b402fac92be70909c8c52" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 33711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:01:52.742655 20323 mvcc.cc:204] Tried to move back new op lower bound from 7195676314557394944 to 7195676314466861056. Current Snapshot: MvccSnapshot[applied={T|T < 7195676314557394944}]
I20250901 20:01:52.750499 19841 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53752:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:52.750694 19841 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53752:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:01:52.752980 19841 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9b13bda4c1174d80a2296a1fdc124ef8: Sending DeleteTablet for 3 replicas of tablet 0fcab025afb047d09352e336af6a9a1c
I20250901 20:01:52.753593 19963 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fcab025afb047d09352e336af6a9a1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:52 UTC) from {username='slave'} at 127.0.0.1:46514
I20250901 20:01:52.753670 20093 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fcab025afb047d09352e336af6a9a1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:52 UTC) from {username='slave'} at 127.0.0.1:38518
I20250901 20:01:52.753867 20225 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fcab025afb047d09352e336af6a9a1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:01:52 UTC) from {username='slave'} at 127.0.0.1:38560
I20250901 20:01:52.754174 20330 tablet_replica.cc:331] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032: stopping tablet replica
I20250901 20:01:52.754410 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 19901
I20250901 20:01:52.754426 20330 raft_consensus.cc:2241] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:01:52.754786 20330 raft_consensus.cc:2270] T 0fcab025afb047d09352e336af6a9a1c P 9bcce3deb1f948a9b22a1219ed7b7032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:52.755234 20331 tablet_replica.cc:331] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52: stopping tablet replica
I20250901 20:01:52.755443 20331 raft_consensus.cc:2241] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:52.755659 20332 tablet_replica.cc:331] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124: stopping tablet replica
I20250901 20:01:52.755717 20331 raft_consensus.cc:2270] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:52.755895 20332 raft_consensus.cc:2241] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:01:52.756500 20332 raft_consensus.cc:2270] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:01:52.757203 20331 ts_tablet_manager.cc:1905] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:52.757583 20332 ts_tablet_manager.cc:1905] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:01:52.759485 20332 ts_tablet_manager.cc:1918] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:01:52.759569 20332 log.cc:1199] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/0fcab025afb047d09352e336af6a9a1c
I20250901 20:01:52.759879 20332 ts_tablet_manager.cc:1939] T 0fcab025afb047d09352e336af6a9a1c P 5fd626b8a1154352bd0be826957b4124: Deleting consensus metadata
I20250901 20:01:52.760317 20331 ts_tablet_manager.cc:1918] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:01:52.760418 20331 log.cc:1199] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/0fcab025afb047d09352e336af6a9a1c
I20250901 20:01:52.760772 20331 ts_tablet_manager.cc:1939] T 0fcab025afb047d09352e336af6a9a1c P 27af2e6d2a6b402fac92be70909c8c52: Deleting consensus metadata
I20250901 20:01:52.760864 19827 catalog_manager.cc:4928] TS 5fd626b8a1154352bd0be826957b4124 (127.14.67.67:42667): tablet 0fcab025afb047d09352e336af6a9a1c (table test-table [id=d17e0a3e8b0c44ddbe73f7a5a3ca282f]) successfully deleted
I20250901 20:01:52.761813 19825 catalog_manager.cc:4928] TS 27af2e6d2a6b402fac92be70909c8c52 (127.14.67.66:33711): tablet 0fcab025afb047d09352e336af6a9a1c (table test-table [id=d17e0a3e8b0c44ddbe73f7a5a3ca282f]) successfully deleted
I20250901 20:01:52.762109 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 20032
W20250901 20:01:52.762092 19825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.67.65:42389 (error 108)
I20250901 20:01:52.763422 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.762841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:42389 (local address 127.0.0.1:46542)
0901 20:01:52.763250 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:52.763323 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.65:42389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277}
W20250901 20:01:52.763550 19825 catalog_manager.cc:4671] TS 9bcce3deb1f948a9b22a1219ed7b7032 (127.14.67.65:42389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fcab025afb047d09352e336af6a9a1c: Network error: Client connection negotiation failed: client connection to 127.14.67.65:42389: connect: Connection refused (error 111)
I20250901 20:01:52.767817 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 20163
I20250901 20:01:52.773212 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 19810
2025-09-01T20:01:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (896 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20250901 20:01:52.794718 14605 mini_oidc.cc:150] Starting JWKS server
I20250901 20:01:52.796120 14605 webserver.cc:480] Webserver started at https://127.0.0.1:35367/ using document root <none> and password file <none>
I20250901 20:01:52.796257 14605 mini_oidc.cc:186] Starting OIDC Discovery server
I20250901 20:01:52.796509 14605 webserver.cc:480] Webserver started at http://127.0.0.1:39663/ using document root <none> and password file <none>
2025-09-01T20:01:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:01:52Z Disabled control of system clock
I20250901 20:01:52.804106 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:46839
--webserver_interface=127.14.67.126
--webserver_port=0
--builtin_ntp_servers=127.14.67.84:42341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:46839
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35367/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/testchainca.cert with env {}
W20250901 20:01:52.904081 20339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:01:52.904290 20339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:01:52.904346 20339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:01:52.907325 20339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:01:52.907372 20339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:01:52.907388 20339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:01:52.907407 20339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:01:52.907424 20339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250901 20:01:52.907441 20339 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250901 20:01:52.907464 20339 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35367/jwks.json
I20250901 20:01:52.911616 20339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:42341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:46839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35367/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:01:52.912600 20339 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:01:52.913360 20339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:01:52.918582 20344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:52.918684 20339 server_base.cc:1047] running on GCE node
W20250901 20:01:52.918623 20347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:01:52.918582 20345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:01:52.919237 20339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:01:52.920065 20339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:01:52.921222 20339 hybrid_clock.cc:648] HybridClock initialized: now 1756756912921190 us; error 39 us; skew 500 ppm
I20250901 20:01:52.922902 20339 webserver.cc:480] Webserver started at http://127.14.67.126:40891/ using document root <none> and password file <none>
I20250901 20:01:52.923410 20339 fs_manager.cc:362] Metadata directory not provided
I20250901 20:01:52.923463 20339 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:01:52.923640 20339 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:01:52.925189 20339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "3f4ffcdafb814337a5ceefd4e4e5ad48"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "a3ffc212c1ccfe2d4bb64a7b7177bbb9"
server_key_iv: "5d7534d60eba34afc5a67b3ee77cd451"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.925639 20339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "3f4ffcdafb814337a5ceefd4e4e5ad48"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "a3ffc212c1ccfe2d4bb64a7b7177bbb9"
server_key_iv: "5d7534d60eba34afc5a67b3ee77cd451"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.928685 20339 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250901 20:01:52.930667 20353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:01:52.931574 20339 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250901 20:01:52.931686 20339 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "3f4ffcdafb814337a5ceefd4e4e5ad48"
format_stamp: "Formatted at 2025-09-01 20:01:52 on dist-test-slave-2s4d"
server_key: "a3ffc212c1ccfe2d4bb64a7b7177bbb9"
server_key_iv: "5d7534d60eba34afc5a67b3ee77cd451"
server_key_version: "encryptionkey@0"
I20250901 20:01:52.931778 20339 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:01:52.948868 20339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:01:52.949344 20339 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:01:52.949491 20339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:01:52.955767 20405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:46839 every 8 connection(s)
I20250901 20:01:52.955765 20339 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:46839
I20250901 20:01:52.956670 20339 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:01:52.958408 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 20339
I20250901 20:01:52.958562 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:01:52.958827 14605 external_mini_cluster.cc:1468] Setting key 89d5e838ebe6d407619c60515b5d9193
I20250901 20:01:52.959188 20406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:01:52.964920 20406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48: Bootstrap starting.
I20250901 20:01:52.966766 20406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48: Neither blocks nor log segments found. Creating new log.
I20250901 20:01:52.967360 20406 log.cc:826] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48: Log is configured to *not* fsync() on all Append() calls
I20250901 20:01:52.968879 20406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48: No bootstrap required, opened a new log
I20250901 20:01:52.970392 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:52.959860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.14.67.126:46839)
0901 20:01:52.960369 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:52.960377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:52.960406 (+ 29us) server_negotiation.cc:407] Connection header received
0901 20:01:52.960968 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:52.960982 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:52.961281 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:52.961514 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:01:52.962095 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:52.962856 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:52.963664 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:52.963846 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:52.968990 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:01:52.969006 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:01:52.969015 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:01:52.969339 (+ 324us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:01:52.969382 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:01:52.969398 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:01:52.969473 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:52.969672 (+ 199us) server_negotiation.cc:299] Negotiation successful
0901 20:01:52.969890 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20250901 20:01:52.971226 20406 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4ffcdafb814337a5ceefd4e4e5ad48" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46839 } }
I20250901 20:01:52.971421 20406 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:01:52.971479 20406 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4ffcdafb814337a5ceefd4e4e5ad48, State: Initialized, Role: FOLLOWER
I20250901 20:01:52.971863 20406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4ffcdafb814337a5ceefd4e4e5ad48" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46839 } }
I20250901 20:01:52.972002 20406 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:01:52.972066 20406 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:01:52.972188 20406 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:01:52.972848 14605 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250901 20:01:52.973065 20406 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4ffcdafb814337a5ceefd4e4e5ad48" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46839 } }
I20250901 20:01:52.973392 20406 leader_election.cc:304] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f4ffcdafb814337a5ceefd4e4e5ad48; no voters:
I20250901 20:01:52.973695 20406 leader_election.cc:290] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:01:52.973769 20411 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:01:52.973965 20411 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [term 1 LEADER]: Becoming Leader. State: Replica: 3f4ffcdafb814337a5ceefd4e4e5ad48, State: Running, Role: LEADER
I20250901 20:01:52.974267 20411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4ffcdafb814337a5ceefd4e4e5ad48" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46839 } }
I20250901 20:01:52.974727 20406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:01:52.975344 20413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f4ffcdafb814337a5ceefd4e4e5ad48. Latest consensus state: current_term: 1 leader_uuid: "3f4ffcdafb814337a5ceefd4e4e5ad48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4ffcdafb814337a5ceefd4e4e5ad48" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46839 } } }
I20250901 20:01:52.975548 20413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:52.975718 20412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f4ffcdafb814337a5ceefd4e4e5ad48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4ffcdafb814337a5ceefd4e4e5ad48" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46839 } } }
I20250901 20:01:52.975871 20412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48 [sys.catalog]: This master's current role is: LEADER
I20250901 20:01:52.976011 20420 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:01:52.978473 20420 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:01:52.983062 20420 catalog_manager.cc:1349] Generated new cluster ID: c5cd7743933641b6af4e6db65e615687
I20250901 20:01:52.983114 20420 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:01:52.987315 20420 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:01:52.988050 20420 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:01:52.997525 20420 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3f4ffcdafb814337a5ceefd4e4e5ad48: Generated new TSK 0
I20250901 20:01:52.998086 20420 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 20:01:53.038089 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:53.009769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.14.67.126:46839)
0901 20:01:53.009888 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:53.009892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:53.010107 (+ 215us) server_negotiation.cc:407] Connection header received
0901 20:01:53.010308 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:53.010310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:53.010362 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:53.010439 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0901 20:01:53.010853 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:53.011439 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:53.012118 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:53.012309 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:53.012395 (+ 86us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0901 20:01:53.037651 (+ 25256us) server_negotiation.cc:377] Sending JWT_EXCHANGE NegotiatePB response
0901 20:01:53.037735 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0901 20:01:53.037905 (+ 170us) server_negotiation.cc:299] Negotiation successful
0901 20:01:53.037949 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34,"thread_start_us":120,"threads_started":1}
I20250901 20:01:53.047401 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:01:53.044136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.14.67.126:46839)
0901 20:01:53.044242 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0901 20:01:53.044246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:01:53.044318 (+ 72us) server_negotiation.cc:407] Connection header received
0901 20:01:53.044446 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:01:53.044449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:01:53.044483 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:01:53.044542 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0901 20:01:53.044937 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:53.045390 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:01:53.046217 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:01:53.046346 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:53.046381 (+ 35us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0901 20:01:53.047164 (+ 783us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0901 20:01:53.047321 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52388: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":46}
W20250901 20:01:53.047458 20409 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52388: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20250901 20:01:53.048214 20442 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:01:53.044007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46839 (local address 127.0.0.1:52388)
0901 20:01:53.044250 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0901 20:01:53.044264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:01:53.044365 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:01:53.044562 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:01:53.044565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:01:53.044573 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0901 20:01:53.044831 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:53.044841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:53.045496 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:01:53.045500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:01:53.046118 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:01:53.046128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:01:53.046236 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:01:53.046256 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0901 20:01:53.047325 (+ 1069us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0901 20:01:53.047471 (+ 146us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.67.126:46839: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20250901 20:02:08.055380 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:08.052409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53912 (local address 127.14.67.126:46839)
0901 20:02:08.052652 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:08.052656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:08.052671 (+ 15us) server_negotiation.cc:407] Connection header received
0901 20:02:08.052732 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:08.052735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:08.052779 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:08.052862 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0901 20:02:08.053184 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.053706 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:08.054352 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.054484 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:08.054545 (+ 61us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0901 20:02:08.055141 (+ 596us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0901 20:02:08.055275 (+ 134us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53912: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
W20250901 20:02:08.055452 20448 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53912: JWT verification failed: token expired
W20250901 20:02:08.055485 20447 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:08.052098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46839 (local address 127.0.0.1:53912)
0901 20:02:08.052402 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:08.052415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:08.052508 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:08.052888 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:08.052890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:08.052896 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0901 20:02:08.053083 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:08.053089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.053819 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:08.053822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:08.054230 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:08.054237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.054326 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:08.054343 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0901 20:02:08.055258 (+ 915us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0901 20:02:08.055369 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.67.126:46839: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":188,"threads_started":1}
W20250901 20:02:08.057660 20448 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:08.057038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.14.67.126:46839)
0901 20:02:08.057181 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:08.057184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:08.057236 (+ 52us) server_negotiation.cc:407] Connection header received
0901 20:02:08.057315 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:08.057317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:08.057352 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:08.057418 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:08.057568 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53920: BlockingRecv error: recv got EOF from 127.0.0.1:53920 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20250901 20:02:08.060896 20458 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.14.67.126:46839
W20250901 20:02:08.061450 20448 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:08.060657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.14.67.126:46839)
0901 20:02:08.060770 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:08.060773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:08.060862 (+ 89us) server_negotiation.cc:407] Connection header received
0901 20:02:08.061047 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:08.061050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:08.061090 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:08.061157 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:08.061354 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53928: BlockingRecv error: recv got EOF from 127.0.0.1:53928 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20250901 20:02:08.061579 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 20339
2025-09-01T20:02:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15307 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20250901 20:02:08.101842 14605 mini_oidc.cc:150] Starting JWKS server
I20250901 20:02:08.102763 14605 webserver.cc:480] Webserver started at https://127.0.0.1:34645/ using document root <none> and password file <none>
I20250901 20:02:08.102849 14605 mini_oidc.cc:186] Starting OIDC Discovery server
I20250901 20:02:08.103049 14605 webserver.cc:480] Webserver started at http://127.0.0.1:44129/ using document root <none> and password file <none>
2025-09-01T20:02:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:02:08Z Disabled control of system clock
I20250901 20:02:08.111997 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:44533
--webserver_interface=127.14.67.126
--webserver_port=0
--builtin_ntp_servers=127.14.67.84:39275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:44533
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34645/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/testchainca.cert with env {}
W20250901 20:02:08.212868 20464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:08.213198 20464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:08.213263 20464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:08.217116 20464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:08.217312 20464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:08.217388 20464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:08.217417 20464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:08.217438 20464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250901 20:02:08.217473 20464 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250901 20:02:08.217490 20464 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34645/jwks.json
I20250901 20:02:08.222704 20464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:39275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:44533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:44533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34645/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:08.223599 20464 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:08.224318 20464 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:08.230104 20470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:08.230104 20469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:08.230152 20464 server_base.cc:1047] running on GCE node
W20250901 20:02:08.230165 20472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:08.230921 20464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:08.231791 20464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:08.232962 20464 hybrid_clock.cc:648] HybridClock initialized: now 1756756928232935 us; error 45 us; skew 500 ppm
I20250901 20:02:08.234771 20464 webserver.cc:480] Webserver started at http://127.14.67.126:33057/ using document root <none> and password file <none>
I20250901 20:02:08.235291 20464 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:08.235352 20464 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:08.235525 20464 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:08.237092 20464 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "69dbd5bd75714f8c94052508d6c7ac47"
format_stamp: "Formatted at 2025-09-01 20:02:08 on dist-test-slave-2s4d"
server_key: "911f45221c2c32592ca63e03db81635c"
server_key_iv: "93634e5d4480fe3173570adf2838e672"
server_key_version: "encryptionkey@0"
I20250901 20:02:08.237542 20464 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "69dbd5bd75714f8c94052508d6c7ac47"
format_stamp: "Formatted at 2025-09-01 20:02:08 on dist-test-slave-2s4d"
server_key: "911f45221c2c32592ca63e03db81635c"
server_key_iv: "93634e5d4480fe3173570adf2838e672"
server_key_version: "encryptionkey@0"
I20250901 20:02:08.241153 20464 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:02:08.243616 20478 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:08.245015 20464 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:02:08.245129 20464 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "69dbd5bd75714f8c94052508d6c7ac47"
format_stamp: "Formatted at 2025-09-01 20:02:08 on dist-test-slave-2s4d"
server_key: "911f45221c2c32592ca63e03db81635c"
server_key_iv: "93634e5d4480fe3173570adf2838e672"
server_key_version: "encryptionkey@0"
I20250901 20:02:08.245234 20464 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:08.259656 20464 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:08.260254 20464 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:08.260429 20464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:08.266959 20464 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:44533
I20250901 20:02:08.266968 20530 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:44533 every 8 connection(s)
I20250901 20:02:08.267851 20464 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:02:08.270175 20531 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:08.275259 20531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47: Bootstrap starting.
I20250901 20:02:08.277364 20531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:08.277673 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 20464
I20250901 20:02:08.277788 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:02:08.278014 14605 external_mini_cluster.cc:1468] Setting key bb356f0836061873068c1429f1ab4976
I20250901 20:02:08.278213 20531 log.cc:826] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:08.280133 20531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47: No bootstrap required, opened a new log
I20250901 20:02:08.282972 20531 raft_consensus.cc:357] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69dbd5bd75714f8c94052508d6c7ac47" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44533 } }
I20250901 20:02:08.283159 20531 raft_consensus.cc:383] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:08.283210 20531 raft_consensus.cc:738] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69dbd5bd75714f8c94052508d6c7ac47, State: Initialized, Role: FOLLOWER
I20250901 20:02:08.283650 20531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69dbd5bd75714f8c94052508d6c7ac47" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44533 } }
I20250901 20:02:08.283769 20531 raft_consensus.cc:397] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:02:08.283830 20531 raft_consensus.cc:491] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:02:08.283939 20531 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:08.284689 20531 raft_consensus.cc:513] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69dbd5bd75714f8c94052508d6c7ac47" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44533 } }
I20250901 20:02:08.284977 20531 leader_election.cc:304] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69dbd5bd75714f8c94052508d6c7ac47; no voters:
I20250901 20:02:08.285228 20531 leader_election.cc:290] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:02:08.285456 20536 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:08.285719 20536 raft_consensus.cc:695] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [term 1 LEADER]: Becoming Leader. State: Replica: 69dbd5bd75714f8c94052508d6c7ac47, State: Running, Role: LEADER
I20250901 20:02:08.286213 20531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:08.286139 20536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69dbd5bd75714f8c94052508d6c7ac47" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44533 } }
I20250901 20:02:08.288151 20538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69dbd5bd75714f8c94052508d6c7ac47. Latest consensus state: current_term: 1 leader_uuid: "69dbd5bd75714f8c94052508d6c7ac47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69dbd5bd75714f8c94052508d6c7ac47" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44533 } } }
I20250901 20:02:08.288295 20538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:08.288405 20537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69dbd5bd75714f8c94052508d6c7ac47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69dbd5bd75714f8c94052508d6c7ac47" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 44533 } } }
I20250901 20:02:08.289326 20537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:08.288746 20551 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:02:08.290688 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:08.278969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60950 (local address 127.14.67.126:44533)
0901 20:02:08.279575 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:08.279585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:08.279622 (+ 37us) server_negotiation.cc:407] Connection header received
0901 20:02:08.280263 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:08.280284 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:08.280615 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:08.280922 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:08.281338 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.282466 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:08.283111 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.283387 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:08.288925 (+ 5538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:08.288944 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:08.288959 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:02:08.289463 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:08.289514 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:02:08.289533 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:08.289632 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:08.289939 (+ 307us) server_negotiation.cc:299] Negotiation successful
0901 20:02:08.290109 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":119,"threads_started":1}
I20250901 20:02:08.293469 14605 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250901 20:02:08.293896 20551 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:02:08.298911 20551 catalog_manager.cc:1349] Generated new cluster ID: 18857142471946a3bfb7690e58275288
I20250901 20:02:08.298967 20551 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:02:08.316704 20551 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:02:08.317566 20551 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:02:08.327013 20551 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 69dbd5bd75714f8c94052508d6c7ac47: Generated new TSK 0
I20250901 20:02:08.327627 20551 catalog_manager.cc:1516] Initializing in-progress tserver states...
E20250901 20:02:08.361733 20561 webserver.cc:553] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20250901 20:02:08.361814 20561 webserver.cc:553] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20250901 20:02:08.362380 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:08.339147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.14.67.126:44533)
0901 20:02:08.339272 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:08.339277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:08.339424 (+ 147us) server_negotiation.cc:407] Connection header received
0901 20:02:08.339584 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:08.339586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:08.339636 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:08.339724 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0901 20:02:08.340089 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.340582 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:08.341217 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.341398 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:08.341476 (+ 78us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0901 20:02:08.361970 (+ 20494us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0901 20:02:08.362240 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60954: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":43}
W20250901 20:02:08.362455 20535 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60954: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20250901 20:02:08.362480 20560 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:08.339060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:44533 (local address 127.0.0.1:60954)
0901 20:02:08.339333 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:08.339345 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:08.339456 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:08.339749 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:08.339752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:08.339761 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0901 20:02:08.339986 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:08.339992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.340720 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:08.340723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:08.341112 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:08.341119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.341217 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:08.341239 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0901 20:02:08.362155 (+ 20916us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0901 20:02:08.362289 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.67.126:44533: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":99,"threads_started":1}
I20250901 20:02:08.363176 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 20464
2025-09-01T20:02:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20250901 20:02:08.402671 14605 mini_oidc.cc:150] Starting JWKS server
I20250901 20:02:08.403571 14605 webserver.cc:480] Webserver started at https://127.0.0.1:44041/ using document root <none> and password file <none>
I20250901 20:02:08.403652 14605 mini_oidc.cc:186] Starting OIDC Discovery server
I20250901 20:02:08.403919 14605 webserver.cc:480] Webserver started at http://127.0.0.1:33419/ using document root <none> and password file <none>
2025-09-01T20:02:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:02:08Z Disabled control of system clock
I20250901 20:02:08.413457 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:46175
--webserver_interface=127.14.67.126
--webserver_port=0
--builtin_ntp_servers=127.14.67.84:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:46175
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44041/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20250901 20:02:08.511888 20567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:08.512101 20567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:08.512147 20567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:08.515144 20567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:08.515197 20567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:08.515213 20567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:08.515231 20567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:08.515247 20567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250901 20:02:08.515268 20567 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250901 20:02:08.515300 20567 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44041/jwks.json
I20250901 20:02:08.518887 20567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:46175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:46175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44041/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:08.519842 20567 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:08.520864 20567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:08.526576 20575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:08.526628 20573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:08.526572 20572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:08.527071 20567 server_base.cc:1047] running on GCE node
I20250901 20:02:08.527679 20567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:08.528726 20567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:08.529915 20567 hybrid_clock.cc:648] HybridClock initialized: now 1756756928529879 us; error 55 us; skew 500 ppm
I20250901 20:02:08.531862 20567 webserver.cc:480] Webserver started at http://127.14.67.126:44589/ using document root <none> and password file <none>
I20250901 20:02:08.532400 20567 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:08.532459 20567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:08.532625 20567 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:08.534215 20567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "4e739cfe1659434dbbc7f7c3efc9f67d"
format_stamp: "Formatted at 2025-09-01 20:02:08 on dist-test-slave-2s4d"
server_key: "ffd8275f2a2835b8c4fa0974a842f554"
server_key_iv: "dec4c608107e8a9792e41e62ded9677c"
server_key_version: "encryptionkey@0"
I20250901 20:02:08.534699 20567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "4e739cfe1659434dbbc7f7c3efc9f67d"
format_stamp: "Formatted at 2025-09-01 20:02:08 on dist-test-slave-2s4d"
server_key: "ffd8275f2a2835b8c4fa0974a842f554"
server_key_iv: "dec4c608107e8a9792e41e62ded9677c"
server_key_version: "encryptionkey@0"
I20250901 20:02:08.537756 20567 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250901 20:02:08.539745 20581 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:08.540809 20567 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:02:08.540916 20567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "4e739cfe1659434dbbc7f7c3efc9f67d"
format_stamp: "Formatted at 2025-09-01 20:02:08 on dist-test-slave-2s4d"
server_key: "ffd8275f2a2835b8c4fa0974a842f554"
server_key_iv: "dec4c608107e8a9792e41e62ded9677c"
server_key_version: "encryptionkey@0"
I20250901 20:02:08.541005 20567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:08.556579 20567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:08.557132 20567 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:08.557301 20567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:08.564544 20567 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:46175
I20250901 20:02:08.564550 20633 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:46175 every 8 connection(s)
I20250901 20:02:08.565457 20567 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:02:08.567818 20634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:08.568162 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 20567
I20250901 20:02:08.568377 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:02:08.568800 14605 external_mini_cluster.cc:1468] Setting key d5f20d7500021f92eed0235e8268df7e
I20250901 20:02:08.575803 20634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d: Bootstrap starting.
I20250901 20:02:08.576696 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:08.570026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.14.67.126:46175)
0901 20:02:08.570603 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:08.570611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:08.570644 (+ 33us) server_negotiation.cc:407] Connection header received
0901 20:02:08.571277 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:08.571291 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:08.571598 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:08.571848 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:08.572684 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.573444 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:08.574314 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.574586 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:08.575227 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:08.575262 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:08.575271 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:02:08.575608 (+ 337us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:08.575644 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:02:08.575658 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:08.575740 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:08.575974 (+ 234us) server_negotiation.cc:299] Negotiation successful
0901 20:02:08.576187 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20250901 20:02:08.577807 20634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:08.578540 20634 log.cc:826] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:08.579097 14605 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250901 20:02:08.580204 20634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d: No bootstrap required, opened a new log
I20250901 20:02:08.582439 20634 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e739cfe1659434dbbc7f7c3efc9f67d" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46175 } }
I20250901 20:02:08.582643 20634 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:08.582705 20634 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e739cfe1659434dbbc7f7c3efc9f67d, State: Initialized, Role: FOLLOWER
I20250901 20:02:08.583062 20634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e739cfe1659434dbbc7f7c3efc9f67d" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46175 } }
I20250901 20:02:08.583180 20634 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:02:08.583240 20634 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:02:08.583346 20634 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:08.584096 20634 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e739cfe1659434dbbc7f7c3efc9f67d" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46175 } }
I20250901 20:02:08.584383 20634 leader_election.cc:304] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e739cfe1659434dbbc7f7c3efc9f67d; no voters:
I20250901 20:02:08.584625 20634 leader_election.cc:290] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:02:08.584723 20639 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:08.584929 20639 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [term 1 LEADER]: Becoming Leader. State: Replica: 4e739cfe1659434dbbc7f7c3efc9f67d, State: Running, Role: LEADER
I20250901 20:02:08.585258 20639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e739cfe1659434dbbc7f7c3efc9f67d" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46175 } }
I20250901 20:02:08.585685 20634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:08.586710 20641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e739cfe1659434dbbc7f7c3efc9f67d. Latest consensus state: current_term: 1 leader_uuid: "4e739cfe1659434dbbc7f7c3efc9f67d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e739cfe1659434dbbc7f7c3efc9f67d" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46175 } } }
I20250901 20:02:08.586839 20641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:08.587186 20648 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:02:08.587266 20640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e739cfe1659434dbbc7f7c3efc9f67d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e739cfe1659434dbbc7f7c3efc9f67d" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46175 } } }
I20250901 20:02:08.587437 20640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:08.589948 20648 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:02:08.594686 20648 catalog_manager.cc:1349] Generated new cluster ID: 10c74d34e02540beb9ec09a6e4498a6d
I20250901 20:02:08.594750 20648 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:02:08.625855 20648 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:02:08.626757 20648 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:02:08.644434 20648 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4e739cfe1659434dbbc7f7c3efc9f67d: Generated new TSK 0
I20250901 20:02:08.645228 20648 catalog_manager.cc:1516] Initializing in-progress tserver states...
E20250901 20:02:08.684839 20664 webserver.cc:553] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20250901 20:02:08.684919 20664 webserver.cc:553] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20250901 20:02:08.686717 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:08.662156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.14.67.126:46175)
0901 20:02:08.662297 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:08.662301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:08.662456 (+ 155us) server_negotiation.cc:407] Connection header received
0901 20:02:08.662641 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:08.662643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:08.662691 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:08.662763 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0901 20:02:08.663174 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.663682 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:08.664560 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.664700 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:08.664791 (+ 91us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0901 20:02:08.686080 (+ 21289us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0901 20:02:08.686499 (+ 419us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35902: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":48}
W20250901 20:02:08.686713 20663 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:08.662071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46175 (local address 127.0.0.1:35902)
0901 20:02:08.662364 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:08.662389 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:08.662498 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:08.662792 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:08.662796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:08.662804 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0901 20:02:08.663075 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:08.663082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.663799 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:08.663803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:08.664446 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:08.664456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.664568 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:08.664590 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0901 20:02:08.686312 (+ 21722us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0901 20:02:08.686451 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.67.126:46175: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":98,"threads_started":1}
W20250901 20:02:08.686803 20637 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35902: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20250901 20:02:08.687520 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 20567
2025-09-01T20:02:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-09-01T20:02:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:02:08Z Disabled control of system clock
I20250901 20:02:08.732980 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:38989
--webserver_interface=127.14.67.126
--webserver_port=0
--builtin_ntp_servers=127.14.67.84:41409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:38989
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/testchainca.cert with env {}
W20250901 20:02:08.833709 20668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:08.833911 20668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:08.833986 20668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:08.836985 20668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:08.837042 20668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:08.837073 20668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:08.837107 20668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:08.837144 20668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250901 20:02:08.837185 20668 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20250901 20:02:08.837213 20668 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20250901 20:02:08.841169 20668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:41409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:38989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:38989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:08.842098 20668 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:08.842882 20668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:08.848665 20674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:08.848665 20673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:08.848680 20676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:08.849067 20668 server_base.cc:1047] running on GCE node
I20250901 20:02:08.849560 20668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:08.850420 20668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:08.851598 20668 hybrid_clock.cc:648] HybridClock initialized: now 1756756928851584 us; error 31 us; skew 500 ppm
I20250901 20:02:08.853246 20668 webserver.cc:480] Webserver started at http://127.14.67.126:44417/ using document root <none> and password file <none>
I20250901 20:02:08.853767 20668 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:08.853821 20668 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:08.853977 20668 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:08.855589 20668 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "a1842d2faa4e4608bc911b99eb069557"
format_stamp: "Formatted at 2025-09-01 20:02:08 on dist-test-slave-2s4d"
server_key: "313fbf9058c083550b53f9a746fa2d46"
server_key_iv: "d8f60e80b8e55f53b8962b767943da8a"
server_key_version: "encryptionkey@0"
I20250901 20:02:08.856022 20668 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "a1842d2faa4e4608bc911b99eb069557"
format_stamp: "Formatted at 2025-09-01 20:02:08 on dist-test-slave-2s4d"
server_key: "313fbf9058c083550b53f9a746fa2d46"
server_key_iv: "d8f60e80b8e55f53b8962b767943da8a"
server_key_version: "encryptionkey@0"
I20250901 20:02:08.859036 20668 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250901 20:02:08.861021 20682 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:08.861987 20668 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:08.862088 20668 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "a1842d2faa4e4608bc911b99eb069557"
format_stamp: "Formatted at 2025-09-01 20:02:08 on dist-test-slave-2s4d"
server_key: "313fbf9058c083550b53f9a746fa2d46"
server_key_iv: "d8f60e80b8e55f53b8962b767943da8a"
server_key_version: "encryptionkey@0"
I20250901 20:02:08.862176 20668 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:08.901346 20668 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:08.901964 20668 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:08.902139 20668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:08.909888 20668 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:38989
I20250901 20:02:08.909955 20734 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:38989 every 8 connection(s)
I20250901 20:02:08.910904 20668 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:02:08.913383 20735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:08.918032 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 20668
I20250901 20:02:08.918161 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:02:08.918390 14605 external_mini_cluster.cc:1468] Setting key 1b1595ba72eaa97f2179d38d6cd0076c
I20250901 20:02:08.918673 20735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557: Bootstrap starting.
I20250901 20:02:08.920928 20735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:08.921602 20735 log.cc:826] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:08.923297 20735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557: No bootstrap required, opened a new log
I20250901 20:02:08.925416 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:08.919542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.14.67.126:38989)
0901 20:02:08.920117 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:08.920128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:08.920158 (+ 30us) server_negotiation.cc:407] Connection header received
0901 20:02:08.920658 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:08.920684 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:08.920986 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:08.921279 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:08.921738 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.922682 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:08.923299 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:08.923573 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:08.924058 (+ 485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:08.924075 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:08.924084 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:02:08.924415 (+ 331us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:08.924450 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:02:08.924471 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:08.924558 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:08.924713 (+ 155us) server_negotiation.cc:299] Negotiation successful
0901 20:02:08.924908 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
I20250901 20:02:08.925853 20735 raft_consensus.cc:357] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1842d2faa4e4608bc911b99eb069557" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38989 } }
I20250901 20:02:08.926024 20735 raft_consensus.cc:383] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:08.926119 20735 raft_consensus.cc:738] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1842d2faa4e4608bc911b99eb069557, State: Initialized, Role: FOLLOWER
I20250901 20:02:08.927016 20735 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1842d2faa4e4608bc911b99eb069557" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38989 } }
I20250901 20:02:08.927151 20735 raft_consensus.cc:397] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:02:08.927253 20735 raft_consensus.cc:491] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:02:08.927367 20735 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:08.927764 14605 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20250901 20:02:08.928265 20735 raft_consensus.cc:513] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1842d2faa4e4608bc911b99eb069557" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38989 } }
I20250901 20:02:08.928606 20735 leader_election.cc:304] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1842d2faa4e4608bc911b99eb069557; no voters:
I20250901 20:02:08.928877 20735 leader_election.cc:290] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:02:08.929137 20740 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:08.929504 20740 raft_consensus.cc:695] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [term 1 LEADER]: Becoming Leader. State: Replica: a1842d2faa4e4608bc911b99eb069557, State: Running, Role: LEADER
I20250901 20:02:08.929960 20740 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1842d2faa4e4608bc911b99eb069557" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38989 } }
I20250901 20:02:08.929976 20735 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:08.932104 20741 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1842d2faa4e4608bc911b99eb069557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1842d2faa4e4608bc911b99eb069557" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38989 } } }
I20250901 20:02:08.932228 20741 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:08.932197 20742 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1842d2faa4e4608bc911b99eb069557. Latest consensus state: current_term: 1 leader_uuid: "a1842d2faa4e4608bc911b99eb069557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1842d2faa4e4608bc911b99eb069557" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 38989 } } }
I20250901 20:02:08.932724 20742 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557 [sys.catalog]: This master's current role is: LEADER
W20250901 20:02:08.935067 20754 catalog_manager.cc:1560] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250901 20:02:08.935139 20754 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250901 20:02:08.935195 20755 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:02:08.935856 20755 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:02:08.940960 20755 catalog_manager.cc:1349] Generated new cluster ID: d7989e64747249dc83d5818f0c4d24e8
I20250901 20:02:08.941036 20755 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:02:08.971760 20755 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:02:08.972824 20755 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:02:08.985385 20755 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a1842d2faa4e4608bc911b99eb069557: Generated new TSK 0
I20250901 20:02:08.986140 20755 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 20:02:09.062440 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:08.998354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.14.67.126:38989)
0901 20:02:08.998478 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:08.998483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:08.998632 (+ 149us) server_negotiation.cc:407] Connection header received
0901 20:02:08.998753 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:08.998756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:08.998814 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:08.998916 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0901 20:02:08.999232 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:09.000045 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:09.000666 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:09.000862 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:09.000953 (+ 91us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0901 20:02:09.061927 (+ 60974us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0901 20:02:09.062255 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36202: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20250901 20:02:09.062489 20764 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:08.998259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:38989 (local address 127.0.0.1:36202)
0901 20:02:08.998542 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:08.998557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:08.998644 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:08.998921 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:08.998923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:08.998929 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0901 20:02:08.999122 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:08.999128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:09.000155 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:09.000158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:09.000558 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:09.000565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:09.000647 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:09.000663 (+ 16us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0901 20:02:09.062168 (+ 61505us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0901 20:02:09.062327 (+ 159us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.67.126:38989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
W20250901 20:02:09.062546 20738 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36202: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
I20250901 20:02:09.063258 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 20668
2025-09-01T20:02:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (356 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (111 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (112 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:02:09 dist-test-slave-2s4d krb5kdc[20776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:02:09 dist-test-slave-2s4d krb5kdc[20776](info): set up 2 sockets
Sep 01 20:02:09 dist-test-slave-2s4d krb5kdc[20776](info): commencing operation
krb5kdc: starting...
W20250901 20:02:11.288105 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.980s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:02:11 dist-test-slave-2s4d krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:02:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:11.431525 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:41417
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:41417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:11.538131 20792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:11.538340 20792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:11.538421 20792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:11.542481 20792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:11.542553 20792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:11.542577 20792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:11.542595 20792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:11.542613 20792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:02:11.547238 20792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:41417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:41417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:11.548524 20792 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:11.549508 20792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:11.556131 20797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:11.556180 20798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:11.556205 20800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:11.557301 20792 server_base.cc:1047] running on GCE node
I20250901 20:02:11.557920 20792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:11.558935 20792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:11.560103 20792 hybrid_clock.cc:648] HybridClock initialized: now 1756756931560083 us; error 38 us; skew 500 ppm
Sep 01 20:02:11 dist-test-slave-2s4d krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756931, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:11.563030 20792 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:02:11.564069 20792 webserver.cc:480] Webserver started at http://127.14.67.126:42025/ using document root <none> and password file <none>
I20250901 20:02:11.564599 20792 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:11.564651 20792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:11.564802 20792 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:11.566452 20792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "2fba63247a9645b781d5766d9d7629fa"
format_stamp: "Formatted at 2025-09-01 20:02:11 on dist-test-slave-2s4d"
server_key: "c708b0d046c8048686d30a437625e181"
server_key_iv: "fac70ce610b01112bd20ae192e344509"
server_key_version: "encryptionkey@0"
I20250901 20:02:11.566955 20792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "2fba63247a9645b781d5766d9d7629fa"
format_stamp: "Formatted at 2025-09-01 20:02:11 on dist-test-slave-2s4d"
server_key: "c708b0d046c8048686d30a437625e181"
server_key_iv: "fac70ce610b01112bd20ae192e344509"
server_key_version: "encryptionkey@0"
I20250901 20:02:11.570933 20792 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250901 20:02:11.573280 20807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:11.574471 20792 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:11.574638 20792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "2fba63247a9645b781d5766d9d7629fa"
format_stamp: "Formatted at 2025-09-01 20:02:11 on dist-test-slave-2s4d"
server_key: "c708b0d046c8048686d30a437625e181"
server_key_iv: "fac70ce610b01112bd20ae192e344509"
server_key_version: "encryptionkey@0"
I20250901 20:02:11.574735 20792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:11.585036 20792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:11.587986 20792 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:11.588136 20792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:11.596916 20792 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:41417
I20250901 20:02:11.596935 20859 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:41417 every 8 connection(s)
I20250901 20:02:11.598096 20792 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:02:11.600621 20860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:11.605815 20860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa: Bootstrap starting.
I20250901 20:02:11.607635 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 20792
I20250901 20:02:11.607774 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:02:11.608016 20860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:11.608067 14605 external_mini_cluster.cc:1468] Setting key ed229afa6ce22eacacf920695c0fcbab
I20250901 20:02:11.608704 20860 log.cc:826] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:11.611049 20860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa: No bootstrap required, opened a new log
I20250901 20:02:11.614588 20860 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fba63247a9645b781d5766d9d7629fa" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 41417 } }
I20250901 20:02:11.614877 20860 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:11.614976 20860 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fba63247a9645b781d5766d9d7629fa, State: Initialized, Role: FOLLOWER
I20250901 20:02:11.615458 20860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fba63247a9645b781d5766d9d7629fa" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 41417 } }
I20250901 20:02:11.615605 20860 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:02:11.615669 20860 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:02:11.615779 20860 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [term 0 FOLLOWER]: Advancing to term 1
Sep 01 20:02:11 dist-test-slave-2s4d krb5kdc[20776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:02:11.616605 20860 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fba63247a9645b781d5766d9d7629fa" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 41417 } }
I20250901 20:02:11.616899 20860 leader_election.cc:304] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fba63247a9645b781d5766d9d7629fa; no voters:
I20250901 20:02:11.617151 20860 leader_election.cc:290] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:02:11.617282 20865 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:11.617496 20865 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [term 1 LEADER]: Becoming Leader. State: Replica: 2fba63247a9645b781d5766d9d7629fa, State: Running, Role: LEADER
I20250901 20:02:11.617782 20865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fba63247a9645b781d5766d9d7629fa" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 41417 } }
I20250901 20:02:11.618247 20860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:11.619496 20866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fba63247a9645b781d5766d9d7629fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fba63247a9645b781d5766d9d7629fa" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 41417 } } }
I20250901 20:02:11.619603 20866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:11.619902 20867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fba63247a9645b781d5766d9d7629fa. Latest consensus state: current_term: 1 leader_uuid: "2fba63247a9645b781d5766d9d7629fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fba63247a9645b781d5766d9d7629fa" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 41417 } } }
I20250901 20:02:11.619974 20867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:11.620262 20874 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:02:11.622279 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:11.609073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40084 (local address 127.14.67.126:41417)
0901 20:02:11.609470 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:11.609479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:11.609513 (+ 34us) server_negotiation.cc:407] Connection header received
0901 20:02:11.609999 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:11.610013 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:11.610280 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:11.610610 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:11.611726 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:11.612499 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:11.613561 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:11.613778 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:11.616496 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:11.616516 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:11.616530 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:11.616561 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:11.619466 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:11.619958 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:11.619963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:11.619968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:11.620081 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:11.620343 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:11.620349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:11.620351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:11.620795 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:11.620991 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:11.621267 (+ 276us) server_negotiation.cc:299] Negotiation successful
0901 20:02:11.621527 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":98,"threads_started":1}
I20250901 20:02:11.623085 20874 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:02:11.628000 20874 catalog_manager.cc:1349] Generated new cluster ID: b0d3d340bde24b8c9206e70181e1c756
I20250901 20:02:11.628065 20874 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:02:11.647094 20874 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:02:11.647862 20874 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:02:11.656806 20874 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2fba63247a9645b781d5766d9d7629fa: Generated new TSK 0
I20250901 20:02:11.657411 20874 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:11.707654 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:41417
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:11.812412 20888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:11.812597 20888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:11.812650 20888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:11.815627 20888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:11.815675 20888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:11.815749 20888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:02:11.819679 20888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:11.820653 20888 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:11.821440 20888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:11.827608 20894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:11.827800 20893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:11.827615 20896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:11.827883 20888 server_base.cc:1047] running on GCE node
I20250901 20:02:11.828498 20888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:11.829221 20888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:11.830412 20888 hybrid_clock.cc:648] HybridClock initialized: now 1756756931830382 us; error 50 us; skew 500 ppm
Sep 01 20:02:11 dist-test-slave-2s4d krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756931, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:11.833230 20888 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:02:11.834157 20888 webserver.cc:480] Webserver started at http://127.14.67.65:45213/ using document root <none> and password file <none>
I20250901 20:02:11.834698 20888 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:11.834753 20888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:11.834918 20888 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:11.836498 20888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "26b0864757f0422ba231aa4fa0044c9b"
format_stamp: "Formatted at 2025-09-01 20:02:11 on dist-test-slave-2s4d"
server_key: "825efd95a0123ff5116eb7250be18e28"
server_key_iv: "432e739dafbc95a6e8d5a2634349b7ba"
server_key_version: "encryptionkey@0"
I20250901 20:02:11.836936 20888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "26b0864757f0422ba231aa4fa0044c9b"
format_stamp: "Formatted at 2025-09-01 20:02:11 on dist-test-slave-2s4d"
server_key: "825efd95a0123ff5116eb7250be18e28"
server_key_iv: "432e739dafbc95a6e8d5a2634349b7ba"
server_key_version: "encryptionkey@0"
I20250901 20:02:11.840227 20888 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:02:11.842801 20903 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:11.844146 20888 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250901 20:02:11.844314 20888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "26b0864757f0422ba231aa4fa0044c9b"
format_stamp: "Formatted at 2025-09-01 20:02:11 on dist-test-slave-2s4d"
server_key: "825efd95a0123ff5116eb7250be18e28"
server_key_iv: "432e739dafbc95a6e8d5a2634349b7ba"
server_key_version: "encryptionkey@0"
I20250901 20:02:11.844437 20888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:11.864946 20888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:11.868676 20888 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:11.868860 20888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:11.869395 20888 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:11.870221 20888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:11.870270 20888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:11.870327 20888 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:11.870362 20888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:11.879295 20888 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:44799
I20250901 20:02:11.879317 21016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:44799 every 8 connection(s)
I20250901 20:02:11.880203 20888 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:02:11.883018 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 20888
I20250901 20:02:11.883134 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:02:11.883373 14605 external_mini_cluster.cc:1468] Setting key a874d7bf8a3815df3b449d0f21cba402
Sep 01 20:02:11 dist-test-slave-2s4d krb5kdc[20776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756931, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
I20250901 20:02:11.894879 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:11.881839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:46685 (local address 127.14.67.126:41417)
0901 20:02:11.882105 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:11.882109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:11.882682 (+ 573us) server_negotiation.cc:407] Connection header received
0901 20:02:11.883550 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:11.883555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:11.883616 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:11.883714 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:11.885752 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:11.886314 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:11.887584 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:11.887766 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:11.890676 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:11.890716 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:11.890721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:11.890769 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:11.892679 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:11.893189 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:11.893195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:11.893200 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:11.893256 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:11.893527 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:11.893532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:11.893536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:11.893795 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:11.893929 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:11.894404 (+ 475us) server_negotiation.cc:299] Negotiation successful
0901 20:02:11.894605 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20250901 20:02:11.895268 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:11.882050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:41417 (local address 127.14.67.65:46685)
0901 20:02:11.882497 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:11.882580 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:11.883349 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:11.884190 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:11.884204 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:11.884734 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:11.885564 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:11.885580 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:11.886447 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:11.886450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:11.887454 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:11.887463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:11.887684 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:11.888504 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:11.888533 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:11.890433 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:11.892829 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:11.892833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:11.892841 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:11.893074 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:11.893355 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:11.893357 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:11.893359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:11.893431 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:11.893920 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:11.893925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:11.894155 (+ 230us) client_negotiation.cc:769] Sending connection context
0901 20:02:11.894334 (+ 179us) client_negotiation.cc:240] Negotiation successful
0901 20:02:11.894621 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":103,"threads_started":1}
I20250901 20:02:11.896649 21017 heartbeater.cc:344] Connected to a master server at 127.14.67.126:41417
I20250901 20:02:11.896958 21017 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
I20250901 20:02:11.897554 21017 heartbeater.cc:507] Master 127.14.67.126:41417 requested a full tablet report, sending...
I20250901 20:02:11.899307 20824 ts_manager.cc:194] Registered new tserver with Master: 26b0864757f0422ba231aa4fa0044c9b (127.14.67.65:44799)
I20250901 20:02:11.900897 20824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:46685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:11.934862 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:41417
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:12.040899 21024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:12.041100 21024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:12.041172 21024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:12.044167 21024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:12.044220 21024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:12.044296 21024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:02:12.048171 21024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:12.049103 21024 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:12.049894 21024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:12.056099 21032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:12.056104 21030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:12.056099 21029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:12.056422 21024 server_base.cc:1047] running on GCE node
I20250901 20:02:12.056773 21024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:12.057312 21024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:12.058460 21024 hybrid_clock.cc:648] HybridClock initialized: now 1756756932058451 us; error 28 us; skew 500 ppm
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:12.061081 21024 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:02:12.062016 21024 webserver.cc:480] Webserver started at http://127.14.67.66:40735/ using document root <none> and password file <none>
I20250901 20:02:12.062553 21024 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:12.062606 21024 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:12.062786 21024 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:12.064399 21024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "1d6fa52ec7e546c39683ac8fb2cf943e"
format_stamp: "Formatted at 2025-09-01 20:02:12 on dist-test-slave-2s4d"
server_key: "de619d4b12cbc591bd59dd9de4c79450"
server_key_iv: "d096219d8a45ec88690acb50796495c3"
server_key_version: "encryptionkey@0"
I20250901 20:02:12.064857 21024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "1d6fa52ec7e546c39683ac8fb2cf943e"
format_stamp: "Formatted at 2025-09-01 20:02:12 on dist-test-slave-2s4d"
server_key: "de619d4b12cbc591bd59dd9de4c79450"
server_key_iv: "d096219d8a45ec88690acb50796495c3"
server_key_version: "encryptionkey@0"
I20250901 20:02:12.067960 21024 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20250901 20:02:12.069957 21039 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:12.070997 21024 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:12.071098 21024 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "1d6fa52ec7e546c39683ac8fb2cf943e"
format_stamp: "Formatted at 2025-09-01 20:02:12 on dist-test-slave-2s4d"
server_key: "de619d4b12cbc591bd59dd9de4c79450"
server_key_iv: "d096219d8a45ec88690acb50796495c3"
server_key_version: "encryptionkey@0"
I20250901 20:02:12.071187 21024 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:12.085897 21024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:12.088950 21024 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:12.089188 21024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:12.089854 21024 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:12.090890 21024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:12.090946 21024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:12.091014 21024 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:12.091043 21024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:12.100898 21024 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:43747
I20250901 20:02:12.101012 21152 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:43747 every 8 connection(s)
I20250901 20:02:12.101831 21024 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:02:12.110049 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 21024
I20250901 20:02:12.110185 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:02:12.110458 14605 external_mini_cluster.cc:1468] Setting key f44bb76138e1efbb9773f7b7ceedbe7a
I20250901 20:02:12.115593 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:12.103604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:40979 (local address 127.14.67.126:41417)
0901 20:02:12.103755 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:12.103760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:12.104412 (+ 652us) server_negotiation.cc:407] Connection header received
0901 20:02:12.105185 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:12.105188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:12.105235 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:12.105310 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:12.106529 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.107041 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:12.108133 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.108272 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:12.110962 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:12.110988 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:12.110990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:12.111022 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:12.113225 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:12.113824 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:12.113831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:12.113836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:12.113888 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:12.114361 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:12.114363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:12.114365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:12.114559 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:12.114651 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:12.115191 (+ 540us) server_negotiation.cc:299] Negotiation successful
0901 20:02:12.115349 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250901 20:02:12.116024 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:12.103865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:41417 (local address 127.14.67.66:40979)
0901 20:02:12.104264 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:12.104310 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:12.104988 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:12.105453 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:12.105460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:12.105820 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:12.106358 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:12.106370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.107156 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:12.107159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:12.108013 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:12.108020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.108178 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:12.108808 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:12.108844 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:12.110719 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:12.113401 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:12.113407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:12.113419 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:12.113710 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:12.114050 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:12.114057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:12.114062 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:12.114216 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:12.114677 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:12.114684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:12.114938 (+ 254us) client_negotiation.cc:769] Sending connection context
0901 20:02:12.115112 (+ 174us) client_negotiation.cc:240] Negotiation successful
0901 20:02:12.115339 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
I20250901 20:02:12.117041 21153 heartbeater.cc:344] Connected to a master server at 127.14.67.126:41417
I20250901 20:02:12.117321 21153 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:12.117962 21153 heartbeater.cc:507] Master 127.14.67.126:41417 requested a full tablet report, sending...
I20250901 20:02:12.119338 20824 ts_manager.cc:194] Registered new tserver with Master: 1d6fa52ec7e546c39683ac8fb2cf943e (127.14.67.66:43747)
I20250901 20:02:12.119985 20824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:40979
WARNING: no policy specified for kudu/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:12.159528 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:41417
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:12.263875 21160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:12.264068 21160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:12.264122 21160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:12.267071 21160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:12.267117 21160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:12.267191 21160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:02:12.271106 21160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:12.272161 21160 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:12.272924 21160 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:12.279153 21168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:12.279143 21165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:12.279301 21160 server_base.cc:1047] running on GCE node
W20250901 20:02:12.279143 21166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:12.279698 21160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:12.280192 21160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:12.281395 21160 hybrid_clock.cc:648] HybridClock initialized: now 1756756932281350 us; error 54 us; skew 500 ppm
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:12.284889 21160 init.cc:377] Logged in from keytab as kudu/127.14.67.67@KRBTEST.COM (short username kudu)
I20250901 20:02:12.286130 21160 webserver.cc:480] Webserver started at http://127.14.67.67:44713/ using document root <none> and password file <none>
I20250901 20:02:12.286798 21160 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:12.286851 21160 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:12.287043 21160 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:12.289006 21160 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "d21e3e7ec15b49569ed55db805b767ca"
format_stamp: "Formatted at 2025-09-01 20:02:12 on dist-test-slave-2s4d"
server_key: "5cb0a7efd928fcea30737a90c02bbb0b"
server_key_iv: "eaf45479618cb5c67dfd06b75464d6eb"
server_key_version: "encryptionkey@0"
I20250901 20:02:12.289489 21160 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "d21e3e7ec15b49569ed55db805b767ca"
format_stamp: "Formatted at 2025-09-01 20:02:12 on dist-test-slave-2s4d"
server_key: "5cb0a7efd928fcea30737a90c02bbb0b"
server_key_iv: "eaf45479618cb5c67dfd06b75464d6eb"
server_key_version: "encryptionkey@0"
I20250901 20:02:12.293349 21160 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250901 20:02:12.295833 21175 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:12.297014 21160 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:02:12.297134 21160 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "d21e3e7ec15b49569ed55db805b767ca"
format_stamp: "Formatted at 2025-09-01 20:02:12 on dist-test-slave-2s4d"
server_key: "5cb0a7efd928fcea30737a90c02bbb0b"
server_key_iv: "eaf45479618cb5c67dfd06b75464d6eb"
server_key_version: "encryptionkey@0"
I20250901 20:02:12.297269 21160 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:12.315032 21160 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:12.317982 21160 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:12.318161 21160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:12.318768 21160 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:12.319626 21160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:12.319674 21160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:12.319722 21160 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:12.319795 21160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:12.328830 21160 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:36339
I20250901 20:02:12.328857 21288 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:36339 every 8 connection(s)
I20250901 20:02:12.329723 21160 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:02:12.334877 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 21160
I20250901 20:02:12.335002 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:02:12.335271 14605 external_mini_cluster.cc:1468] Setting key 769a8dc5f302d6c01a5950baea019121
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:02:12.341444 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:12.331395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:53449 (local address 127.14.67.126:41417)
0901 20:02:12.331529 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:12.331533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:12.332116 (+ 583us) server_negotiation.cc:407] Connection header received
0901 20:02:12.332928 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:12.332931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:12.332993 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:12.333070 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:12.334254 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.334855 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:12.336145 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.336326 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:12.338550 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:12.338564 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:12.338566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:12.338588 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:12.339782 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:12.340224 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:12.340228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:12.340229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:12.340269 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:12.340514 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:12.340517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:12.340519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:12.340652 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:12.340738 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:12.341175 (+ 437us) server_negotiation.cc:299] Negotiation successful
0901 20:02:12.341286 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250901 20:02:12.342010 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:12.331587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:41417 (local address 127.14.67.67:53449)
0901 20:02:12.331982 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:12.332015 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:12.332732 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:12.333233 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:12.333240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:12.333599 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:12.334114 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:12.334126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.335026 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:12.335030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:12.336018 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:12.336028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.336219 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:12.336865 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:12.336895 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:12.338392 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:12.339890 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:12.339896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:12.339906 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:12.340127 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:12.340348 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:12.340353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:12.340356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:12.340430 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:12.340748 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:12.340756 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:12.340950 (+ 194us) client_negotiation.cc:769] Sending connection context
0901 20:02:12.341126 (+ 176us) client_negotiation.cc:240] Negotiation successful
0901 20:02:12.341380 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
I20250901 20:02:12.343384 21289 heartbeater.cc:344] Connected to a master server at 127.14.67.126:41417
I20250901 20:02:12.343742 21289 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:12.344449 21289 heartbeater.cc:507] Master 127.14.67.126:41417 requested a full tablet report, sending...
I20250901 20:02:12.345769 20824 ts_manager.cc:194] Registered new tserver with Master: d21e3e7ec15b49569ed55db805b767ca (127.14.67.67:36339)
I20250901 20:02:12.346421 20824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:53449
I20250901 20:02:12.349238 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/test-admin.keytab.
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:12.365665 14605 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20250901 20:02:12.368953 21296 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:12.367166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.14.67.65:44799)
0901 20:02:12.367537 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:12.367544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:12.367571 (+ 27us) server_negotiation.cc:407] Connection header received
0901 20:02:12.367646 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:12.367652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:12.367864 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:12.368037 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:12.368678 (+ 641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53678: BlockingRecv error: recv got EOF from 127.0.0.1:53678 (error 108)
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":141,"threads_started":1}
W20250901 20:02:12.369038 14605 init.cc:244] Error while opening credential cache '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/krb5cc)
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:12.370286 14605 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:12.371575 14605 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.65@KRBTEST.COM
I20250901 20:02:12.381834 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:12.372890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.14.67.65:44799)
0901 20:02:12.373018 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:12.373022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:12.373062 (+ 40us) server_negotiation.cc:407] Connection header received
0901 20:02:12.373213 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:12.373216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:12.373262 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:12.373336 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:12.373955 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.374762 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:12.375536 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.375741 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:12.378041 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:12.378065 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:12.378077 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:12.378101 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:12.379958 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:12.380347 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:12.380352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:12.380353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:12.380404 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:12.380805 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:12.380808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:12.380809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:12.381019 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:12.381227 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:12.381470 (+ 243us) server_negotiation.cc:299] Negotiation successful
0901 20:02:12.381603 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20250901 20:02:12.385924 21296 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:12.385129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.14.67.65:44799)
0901 20:02:12.385261 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:12.385264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:12.385320 (+ 56us) server_negotiation.cc:407] Connection header received
0901 20:02:12.385411 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:12.385413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:12.385452 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:12.385524 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:12.385835 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53700: BlockingRecv error: recv got EOF from 127.0.0.1:53700 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20250901 20:02:12.386049 14605 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/krb5cc' of type FILE
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:12.387079 14605 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:12.387884 14605 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.65@KRBTEST.COM
I20250901 20:02:12.395344 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:12.388659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.14.67.65:44799)
0901 20:02:12.388810 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:12.388817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:12.388929 (+ 112us) server_negotiation.cc:407] Connection header received
0901 20:02:12.388972 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:12.388974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:12.389025 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:12.389097 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:12.389587 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.390052 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:12.390647 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.390773 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:12.392522 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:12.392535 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:12.392538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:12.392559 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:12.393982 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:12.394373 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:12.394376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:12.394378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:12.394430 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:12.394722 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:12.394725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:12.394726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:12.394844 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:12.394967 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:12.395093 (+ 126us) server_negotiation.cc:299] Negotiation successful
0901 20:02:12.395176 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20250901 20:02:12.397704 21296 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:12.396886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.14.67.65:44799)
0901 20:02:12.396988 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:12.396993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:12.397055 (+ 62us) server_negotiation.cc:407] Connection header received
0901 20:02:12.397184 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:12.397189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:12.397221 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:12.397284 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:12.397633 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53708: BlockingRecv error: recv got EOF from 127.0.0.1:53708 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20250901 20:02:12.397837 14605 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1756756873705473-14605-0/krb5kdc/krb5cc' of type FILE
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:12.398759 14605 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:12.399542 14605 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[20776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.65@KRBTEST.COM
I20250901 20:02:12.407452 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:12.400205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53710 (local address 127.14.67.65:44799)
0901 20:02:12.400323 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:12.400326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:12.400387 (+ 61us) server_negotiation.cc:407] Connection header received
0901 20:02:12.400463 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:12.400465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:12.400495 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:12.400556 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:12.401042 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.401524 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:12.402102 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:12.402224 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:12.404258 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:12.404284 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:12.404287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:12.404325 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:12.405929 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:12.406407 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:12.406410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:12.406411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:12.406450 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:12.406785 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:12.406788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:12.406790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:12.406901 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:12.406999 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:12.407266 (+ 267us) server_negotiation.cc:299] Negotiation successful
0901 20:02:12.407344 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250901 20:02:12.408648 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 20888
I20250901 20:02:12.415860 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 21024
I20250901 20:02:12.421792 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 21160
I20250901 20:02:12.426849 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 20792
2025-09-01T20:02:12Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3140 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[21309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[21309](info): set up 2 sockets
Sep 01 20:02:12 dist-test-slave-2s4d krb5kdc[21309](info): commencing operation
krb5kdc: starting...
W20250901 20:02:14.419463 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.971s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:02:14 dist-test-slave-2s4d krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:02:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:02:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
I20250901 20:02:14.555372 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:39725
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:34599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:39725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:14.655369 21325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:14.655694 21325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:14.655798 21325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:14.659796 21325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:14.659894 21325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:14.659945 21325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:14.659982 21325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:14.660027 21325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:02:14.665225 21325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:34599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:39725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:39725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:14.666234 21325 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:14.667059 21325 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:14.672250 21333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:14.672232 21331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:14.672374 21325 server_base.cc:1047] running on GCE node
W20250901 20:02:14.672232 21330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:14.672899 21325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:14.673692 21325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:14.674872 21325 hybrid_clock.cc:648] HybridClock initialized: now 1756756934674858 us; error 28 us; skew 500 ppm
Sep 01 20:02:14 dist-test-slave-2s4d krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756934, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:14.677330 21325 init.cc:377] Logged in from keytab as oryx/127.14.67.126@KRBTEST.COM (short username oryx)
I20250901 20:02:14.678373 21325 webserver.cc:480] Webserver started at http://127.14.67.126:33115/ using document root <none> and password file <none>
I20250901 20:02:14.678961 21325 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:14.679028 21325 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:14.679212 21325 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:14.680961 21325 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "92c3ebb531304adab92f7135e8afbb43"
format_stamp: "Formatted at 2025-09-01 20:02:14 on dist-test-slave-2s4d"
server_key: "779579963fa03686245552821f01cc03"
server_key_iv: "25f76edacaabc3097bfb681794c1a398"
server_key_version: "encryptionkey@0"
I20250901 20:02:14.681458 21325 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "92c3ebb531304adab92f7135e8afbb43"
format_stamp: "Formatted at 2025-09-01 20:02:14 on dist-test-slave-2s4d"
server_key: "779579963fa03686245552821f01cc03"
server_key_iv: "25f76edacaabc3097bfb681794c1a398"
server_key_version: "encryptionkey@0"
I20250901 20:02:14.684783 21325 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:02:14.686926 21340 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:14.687844 21325 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:14.687952 21325 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "92c3ebb531304adab92f7135e8afbb43"
format_stamp: "Formatted at 2025-09-01 20:02:14 on dist-test-slave-2s4d"
server_key: "779579963fa03686245552821f01cc03"
server_key_iv: "25f76edacaabc3097bfb681794c1a398"
server_key_version: "encryptionkey@0"
I20250901 20:02:14.688047 21325 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:14.715186 21325 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:14.722718 21325 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:14.722884 21325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:14.729907 21325 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:39725
I20250901 20:02:14.729928 21392 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:39725 every 8 connection(s)
I20250901 20:02:14.730860 21325 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:02:14.731144 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 21325
I20250901 20:02:14.731264 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:02:14.731531 14605 external_mini_cluster.cc:1468] Setting key 5dbf53bc158a1cac0e7f78a8352be629
I20250901 20:02:14.733747 21393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 01 20:02:14 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:14.739353 21393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43: Bootstrap starting.
I20250901 20:02:14.741588 21393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:14.742331 21393 log.cc:826] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:14.744056 21393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43: No bootstrap required, opened a new log
I20250901 20:02:14.744674 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:14.732685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.14.67.126:39725)
0901 20:02:14.733071 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:14.733081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:14.733118 (+ 37us) server_negotiation.cc:407] Connection header received
0901 20:02:14.733908 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:14.733929 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:14.734329 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:14.734686 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:14.735410 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:14.736546 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:14.737210 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:14.737480 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:14.739616 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:14.739652 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:14.739661 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:14.739690 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:14.741355 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:14.741926 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:14.741934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:14.741940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:14.742077 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:14.742408 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:14.742414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:14.742416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:14.742931 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:14.743192 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:14.743527 (+ 335us) server_negotiation.cc:299] Negotiation successful
0901 20:02:14.743838 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20250901 20:02:14.746549 21393 raft_consensus.cc:357] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } }
I20250901 20:02:14.746745 21393 raft_consensus.cc:383] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:14.746832 21393 raft_consensus.cc:738] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c3ebb531304adab92f7135e8afbb43, State: Initialized, Role: FOLLOWER
I20250901 20:02:14.747203 21393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } }
I20250901 20:02:14.747323 21393 raft_consensus.cc:397] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:02:14.747359 21393 raft_consensus.cc:491] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:02:14.747467 21393 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:14.748193 21393 raft_consensus.cc:513] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } }
I20250901 20:02:14.748484 21393 leader_election.cc:304] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92c3ebb531304adab92f7135e8afbb43; no voters:
I20250901 20:02:14.748718 21393 leader_election.cc:290] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:02:14.748816 21398 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:14.749038 21398 raft_consensus.cc:695] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 1 LEADER]: Becoming Leader. State: Replica: 92c3ebb531304adab92f7135e8afbb43, State: Running, Role: LEADER
I20250901 20:02:14.749317 21398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } }
I20250901 20:02:14.749682 21393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:14.750690 21399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } }
I20250901 20:02:14.750802 21399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:14.751204 21407 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:02:14.751855 21400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92c3ebb531304adab92f7135e8afbb43. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } }
I20250901 20:02:14.751964 21400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:14.753901 21407 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:02:14.758581 21407 catalog_manager.cc:1349] Generated new cluster ID: edd92a8d72a04b938d695a74fda08800
I20250901 20:02:14.758646 21407 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:02:14.768093 21407 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:02:14.769030 21407 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:02:14.774541 21407 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43: Generated new TSK 0
I20250901 20:02:14.775205 21407 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
I20250901 20:02:14.835486 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:39725
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:34599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:14.940706 21421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:14.940910 21421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:14.940965 21421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:14.944129 21421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:14.944185 21421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:14.944259 21421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:02:14.948153 21421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:34599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:39725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:14.949106 21421 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:14.949908 21421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:14.955940 21427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:14.955971 21426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:14.956166 21421 server_base.cc:1047] running on GCE node
W20250901 20:02:14.956288 21429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:14.956568 21421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:14.957028 21421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:14.958223 21421 hybrid_clock.cc:648] HybridClock initialized: now 1756756934958209 us; error 26 us; skew 500 ppm
Sep 01 20:02:14 dist-test-slave-2s4d krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756934, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:14.960664 21421 init.cc:377] Logged in from keytab as oryx/127.14.67.65@KRBTEST.COM (short username oryx)
I20250901 20:02:14.961704 21421 webserver.cc:480] Webserver started at http://127.14.67.65:40877/ using document root <none> and password file <none>
I20250901 20:02:14.962208 21421 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:14.962261 21421 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:14.962421 21421 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:14.964073 21421 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "7d65920bf7264cb5b463e297265d12c4"
format_stamp: "Formatted at 2025-09-01 20:02:14 on dist-test-slave-2s4d"
server_key: "7817388fa48ef330a70bdf8c46220b27"
server_key_iv: "7be44429036a338f3bcc588ce6377aa4"
server_key_version: "encryptionkey@0"
I20250901 20:02:14.964510 21421 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "7d65920bf7264cb5b463e297265d12c4"
format_stamp: "Formatted at 2025-09-01 20:02:14 on dist-test-slave-2s4d"
server_key: "7817388fa48ef330a70bdf8c46220b27"
server_key_iv: "7be44429036a338f3bcc588ce6377aa4"
server_key_version: "encryptionkey@0"
I20250901 20:02:14.967499 21421 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:02:14.969388 21436 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:14.970489 21421 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:14.970631 21421 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "7d65920bf7264cb5b463e297265d12c4"
format_stamp: "Formatted at 2025-09-01 20:02:14 on dist-test-slave-2s4d"
server_key: "7817388fa48ef330a70bdf8c46220b27"
server_key_iv: "7be44429036a338f3bcc588ce6377aa4"
server_key_version: "encryptionkey@0"
I20250901 20:02:14.970713 21421 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:14.986639 21421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:14.990154 21421 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:14.990363 21421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:14.990938 21421 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:14.991923 21421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:14.991983 21421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:14.992048 21421 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:14.992080 21421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:15.001219 21421 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:40197
I20250901 20:02:15.001273 21549 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:40197 every 8 connection(s)
I20250901 20:02:15.002131 21421 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:02:15.011173 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 21421
I20250901 20:02:15.011349 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:02:15.011622 14605 external_mini_cluster.cc:1468] Setting key 523d12a58ea4d91a8d21f5a66c08210d
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756934, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:15.016278 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.004148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:51647 (local address 127.14.67.126:39725)
0901 20:02:15.004364 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.004368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.005016 (+ 648us) server_negotiation.cc:407] Connection header received
0901 20:02:15.006064 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.006069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.006152 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.006240 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.008367 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.008883 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.009536 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.009705 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.012308 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.012326 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.012328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.012351 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.013890 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.014364 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.014367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.014369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.014408 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.014700 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.014706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.014710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.014915 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.015037 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.015889 (+ 852us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.016040 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20250901 20:02:15.017019 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.004270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.65:51647)
0901 20:02:15.004803 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:15.004878 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:15.005832 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:15.006440 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:15.006449 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:15.006969 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:15.008141 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.008165 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.009021 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.009024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:15.009424 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.009431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.009620 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.010414 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:15.010437 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:15.012161 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:15.014004 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.014010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.014020 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.014250 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.014487 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.014492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.014494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.014611 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.015138 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:15.015147 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:15.015529 (+ 382us) client_negotiation.cc:769] Sending connection context
0901 20:02:15.015802 (+ 273us) client_negotiation.cc:240] Negotiation successful
0901 20:02:15.016215 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":109,"threads_started":1}
I20250901 20:02:15.018157 21550 heartbeater.cc:344] Connected to a master server at 127.14.67.126:39725
I20250901 20:02:15.018435 21550 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:15.019134 21550 heartbeater.cc:507] Master 127.14.67.126:39725 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.14.67.66@KRBTEST.COM; defaulting to no policy
I20250901 20:02:15.020951 21357 ts_manager.cc:194] Registered new tserver with Master: 7d65920bf7264cb5b463e297265d12c4 (127.14.67.65:40197)
I20250901 20:02:15.022480 21357 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:51647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
I20250901 20:02:15.060191 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:39725
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:34599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:15.159595 21557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:15.159782 21557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:15.159861 21557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:15.162943 21557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:15.163003 21557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:15.163110 21557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:02:15.166906 21557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:34599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:39725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:15.167809 21557 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:15.168556 21557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:15.174424 21562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:15.174499 21563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:15.174432 21565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:15.174691 21557 server_base.cc:1047] running on GCE node
I20250901 20:02:15.175165 21557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:15.175660 21557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:15.176807 21557 hybrid_clock.cc:648] HybridClock initialized: now 1756756935176782 us; error 41 us; skew 500 ppm
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:15.179338 21557 init.cc:377] Logged in from keytab as oryx/127.14.67.66@KRBTEST.COM (short username oryx)
I20250901 20:02:15.180244 21557 webserver.cc:480] Webserver started at http://127.14.67.66:40739/ using document root <none> and password file <none>
I20250901 20:02:15.180740 21557 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:15.180795 21557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:15.180958 21557 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:15.182610 21557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "05d0551bc0f04bdd8fad024b9196fc0d"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "5958e31dc4071729d31534c06ce015e6"
server_key_iv: "1d29797e6c503844cc6d765fb661f183"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.183050 21557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "05d0551bc0f04bdd8fad024b9196fc0d"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "5958e31dc4071729d31534c06ce015e6"
server_key_iv: "1d29797e6c503844cc6d765fb661f183"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.186044 21557 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250901 20:02:15.188020 21572 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:15.188882 21557 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:02:15.188980 21557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "05d0551bc0f04bdd8fad024b9196fc0d"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "5958e31dc4071729d31534c06ce015e6"
server_key_iv: "1d29797e6c503844cc6d765fb661f183"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.189069 21557 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:15.213764 21557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:15.216786 21557 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:15.216967 21557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:15.217469 21557 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:15.218567 21557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:15.218617 21557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:15.218680 21557 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:15.218716 21557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:15.227420 21557 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:35033
I20250901 20:02:15.227455 21685 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:35033 every 8 connection(s)
I20250901 20:02:15.228627 21557 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:02:15.236138 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 21557
I20250901 20:02:15.236299 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:02:15.236608 14605 external_mini_cluster.cc:1468] Setting key 7372c937ee2d3d03f93f1eea46ca3fcc
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:15.243399 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.230392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:44695 (local address 127.14.67.126:39725)
0901 20:02:15.230558 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.230562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.231369 (+ 807us) server_negotiation.cc:407] Connection header received
0901 20:02:15.232237 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.232242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.232342 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.232479 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.234649 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.235554 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.236489 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.236730 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.239360 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.239375 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.239378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.239400 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.241122 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.241777 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.241780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.241781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.241831 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.242181 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.242183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.242184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.242341 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.242419 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.243110 (+ 691us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.243253 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250901 20:02:15.244217 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.230711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.66:44695)
0901 20:02:15.231198 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:15.231239 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:15.232002 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:15.232705 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:15.232717 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:15.233217 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:15.234464 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.234481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.235724 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.235728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:15.236346 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.236355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.236547 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.237330 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:15.237387 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:15.239181 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:15.241243 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.241251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.241264 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.241630 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.241934 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.241939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.241943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.242080 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.242439 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:15.242446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:15.242756 (+ 310us) client_negotiation.cc:769] Sending connection context
0901 20:02:15.242999 (+ 243us) client_negotiation.cc:240] Negotiation successful
0901 20:02:15.243311 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":105,"threads_started":1}
I20250901 20:02:15.245610 21686 heartbeater.cc:344] Connected to a master server at 127.14.67.126:39725
I20250901 20:02:15.245986 21686 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for oryx/127.14.67.67@KRBTEST.COM; defaulting to no policy
I20250901 20:02:15.246729 21686 heartbeater.cc:507] Master 127.14.67.126:39725 requested a full tablet report, sending...
I20250901 20:02:15.247968 21357 ts_manager.cc:194] Registered new tserver with Master: 05d0551bc0f04bdd8fad024b9196fc0d (127.14.67.66:35033)
I20250901 20:02:15.248701 21357 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:44695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
I20250901 20:02:15.289425 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:39725
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:34599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:15.389829 21693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:15.390007 21693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:15.390074 21693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:15.393153 21693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:15.393208 21693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:15.393288 21693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:02:15.397123 21693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:34599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:39725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:15.398046 21693 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:15.398806 21693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:15.405311 21701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:15.405354 21699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:15.405580 21693 server_base.cc:1047] running on GCE node
W20250901 20:02:15.405354 21698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:15.406056 21693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:15.406770 21693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:15.407950 21693 hybrid_clock.cc:648] HybridClock initialized: now 1756756935407929 us; error 43 us; skew 500 ppm
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:15.410452 21693 init.cc:377] Logged in from keytab as oryx/127.14.67.67@KRBTEST.COM (short username oryx)
I20250901 20:02:15.411429 21693 webserver.cc:480] Webserver started at http://127.14.67.67:38159/ using document root <none> and password file <none>
I20250901 20:02:15.411932 21693 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:15.411986 21693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:15.412163 21693 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:15.413758 21693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "39db649d1bd845a0ac3324d3361819ea"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "c98f8abaa5b3ee9e3ab3d282403f8520"
server_key_iv: "d9ba1f015a70bf4a1599cff704f41886"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.414204 21693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "39db649d1bd845a0ac3324d3361819ea"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "c98f8abaa5b3ee9e3ab3d282403f8520"
server_key_iv: "d9ba1f015a70bf4a1599cff704f41886"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.417253 21693 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20250901 20:02:15.419198 21708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:15.420358 21693 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250901 20:02:15.420464 21693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "39db649d1bd845a0ac3324d3361819ea"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "c98f8abaa5b3ee9e3ab3d282403f8520"
server_key_iv: "d9ba1f015a70bf4a1599cff704f41886"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.420552 21693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:15.430590 21693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:15.433202 21693 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:15.433367 21693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:15.433835 21693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:15.434630 21693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:15.434682 21693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:15.434739 21693 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:15.434772 21693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:15.442953 21693 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:42619
I20250901 20:02:15.442982 21821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:42619 every 8 connection(s)
I20250901 20:02:15.443779 21693 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:02:15.445544 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 21693
I20250901 20:02:15.445647 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:02:15.445894 14605 external_mini_cluster.cc:1468] Setting key e3a5a0908f99c4b41099f8a86a15af0a
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:15.455061 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.445330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:45007 (local address 127.14.67.126:39725)
0901 20:02:15.445502 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.445507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.446067 (+ 560us) server_negotiation.cc:407] Connection header received
0901 20:02:15.447006 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.447010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.447065 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.447143 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.448744 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.449400 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.449960 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.450098 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.452074 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.452087 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.452089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.452110 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.453461 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.453853 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.453856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.453857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.453896 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.454098 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.454100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.454101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.454222 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.454295 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.454837 (+ 542us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.454935 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20250901 20:02:15.455560 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.445518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.67:45007)
0901 20:02:15.445932 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:15.445973 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:15.446836 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:15.447267 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:15.447274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:15.447656 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:15.448522 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.448534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.449494 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.449496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:15.449866 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.449871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.450001 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.450489 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:15.450540 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:15.451927 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:15.453556 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.453560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.453567 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.453764 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.453966 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.453968 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.453969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.454034 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.454296 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:15.454300 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:15.454591 (+ 291us) client_negotiation.cc:769] Sending connection context
0901 20:02:15.454779 (+ 188us) client_negotiation.cc:240] Negotiation successful
0901 20:02:15.454963 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":136,"threads_started":1}
I20250901 20:02:15.456442 21822 heartbeater.cc:344] Connected to a master server at 127.14.67.126:39725
I20250901 20:02:15.456653 21822 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:15.457101 21822 heartbeater.cc:507] Master 127.14.67.126:39725 requested a full tablet report, sending...
I20250901 20:02:15.458047 21357 ts_manager.cc:194] Registered new tserver with Master: 39db649d1bd845a0ac3324d3361819ea (127.14.67.67:42619)
I20250901 20:02:15.458750 21357 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:45007
I20250901 20:02:15.459267 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.67.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.67.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.67.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.67.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
I20250901 20:02:15.510080 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.125:45013
--webserver_interface=127.14.67.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:34599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:39725,127.14.67.125:45013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:15.609975 21829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:15.610163 21829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:15.610213 21829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:15.613269 21829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:15.613325 21829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:15.613348 21829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:15.613366 21829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:15.613384 21829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:02:15.617296 21829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:34599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:39725,127.14.67.125:45013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.125:45013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.67.125
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
W20250901 20:02:15.617489 21829 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.67.126:39725,127.14.67.125:45013'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20250901 20:02:15.618278 21829 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:15.619077 21829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:15.624125 21834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:15.624195 21835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:15.624159 21837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:15.624316 21829 server_base.cc:1047] running on GCE node
I20250901 20:02:15.624745 21829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:15.625593 21829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:15.626762 21829 hybrid_clock.cc:648] HybridClock initialized: now 1756756935626753 us; error 23 us; skew 500 ppm
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:15.629473 21829 init.cc:377] Logged in from keytab as oryx/127.14.67.125@KRBTEST.COM (short username oryx)
I20250901 20:02:15.630553 21829 webserver.cc:480] Webserver started at http://127.14.67.125:34489/ using document root <none> and password file <none>
I20250901 20:02:15.631050 21829 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:15.631104 21829 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:15.631265 21829 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:15.632833 21829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data/instance:
uuid: "5f0800e54af54372b1e66ac2c3f428ea"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "327969d131fae26fea1dfb6f690ab8c8"
server_key_iv: "54cbdb21b1a14997c767f45339e24e8c"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.633267 21829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/wal/instance:
uuid: "5f0800e54af54372b1e66ac2c3f428ea"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "327969d131fae26fea1dfb6f690ab8c8"
server_key_iv: "54cbdb21b1a14997c767f45339e24e8c"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.636291 21829 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:02:15.638214 21844 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:15.639242 21829 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:15.639341 21829 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/wal
uuid: "5f0800e54af54372b1e66ac2c3f428ea"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "327969d131fae26fea1dfb6f690ab8c8"
server_key_iv: "54cbdb21b1a14997c767f45339e24e8c"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.639432 21829 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:15.656903 21829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:15.659678 21829 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:15.659855 21829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:15.667135 21829 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.125:45013
I20250901 20:02:15.667117 21896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.125:45013 every 8 connection(s)
I20250901 20:02:15.668115 21829 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data/info.pb
I20250901 20:02:15.670480 21897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:15.674144 21897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } has no permanent_uuid. Determining permanent_uuid...
I20250901 20:02:15.675493 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 21829
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.125@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.67.125@KRBTEST.COM
I20250901 20:02:15.686600 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.675930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.14.67.126:39725)
0901 20:02:15.676121 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.676125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.676957 (+ 832us) server_negotiation.cc:407] Connection header received
0901 20:02:15.677635 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.677638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.677694 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.677770 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.678787 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.679288 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.679914 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.680114 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.682350 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.682370 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.682373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.682399 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.684119 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.684775 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.684779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.684781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.684877 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.685334 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.685339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.685341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.685563 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.685668 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.686226 (+ 558us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.686389 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20250901 20:02:15.687444 21901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.676972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.14.67.125:45013)
0901 20:02:15.677248 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.677253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.677283 (+ 30us) server_negotiation.cc:407] Connection header received
0901 20:02:15.677476 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.677491 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.677649 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.677810 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.678614 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.679363 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.679997 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.680245 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.682840 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.682856 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.682864 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.682884 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.684814 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.685252 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.685256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.685258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.685319 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.685553 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.685557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.685559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.685876 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.686078 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.686299 (+ 221us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.686545 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20250901 20:02:15.687455 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.676212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:35910)
0901 20:02:15.676808 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:15.676852 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:15.677462 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:15.677839 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:15.677844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:15.678303 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:15.678640 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.678656 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.679394 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.679400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:15.679801 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.679808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.679978 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.680755 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:15.680777 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:15.682213 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:15.684244 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.684252 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.684263 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.684561 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.685078 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.685082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.685084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.685226 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.685685 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:15.685700 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:15.685893 (+ 193us) client_negotiation.cc:769] Sending connection context
0901 20:02:15.686182 (+ 289us) client_negotiation.cc:240] Negotiation successful
0901 20:02:15.686661 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":141,"thread_start_us":140,"threads_started":1}
I20250901 20:02:15.689637 21897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } has no permanent_uuid. Determining permanent_uuid...
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.125@KRBTEST.COM for oryx/127.14.67.125@KRBTEST.COM
I20250901 20:02:15.697098 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.690176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.125:45013 (local address 127.0.0.1:53098)
0901 20:02:15.690301 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:15.690315 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:15.690390 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:15.690899 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:15.690903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:15.691071 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:15.691226 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.691231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.691876 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.691880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:15.692251 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.692282 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.692380 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.692824 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:15.692835 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:15.694001 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:15.696016 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.696019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.696020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.696200 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.696439 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.696441 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.696442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.696477 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.696727 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:15.696729 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:15.696792 (+ 63us) client_negotiation.cc:769] Sending connection context
0901 20:02:15.696872 (+ 80us) client_negotiation.cc:240] Negotiation successful
0901 20:02:15.696974 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20250901 20:02:15.697201 21901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.690460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.14.67.125:45013)
0901 20:02:15.690687 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.690690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.690705 (+ 15us) server_negotiation.cc:407] Connection header received
0901 20:02:15.690739 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.690742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.690780 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.690875 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.691348 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.691768 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.692369 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.692570 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.694126 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.694139 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.694141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.694161 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.695826 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.696314 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.696317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.696318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.696361 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.696538 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.696540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.696541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.696658 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.696727 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.696909 (+ 182us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.697047 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":135}
I20250901 20:02:15.700387 21897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: Bootstrap starting.
I20250901 20:02:15.703289 21897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:15.704128 21897 log.cc:826] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:15.706183 21897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: No bootstrap required, opened a new log
I20250901 20:02:15.708855 21897 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } }
I20250901 20:02:15.709035 21897 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:15.709071 21897 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0800e54af54372b1e66ac2c3f428ea, State: Initialized, Role: FOLLOWER
I20250901 20:02:15.709547 21897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } }
I20250901 20:02:15.710071 21903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } } }
I20250901 20:02:15.710225 21903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: This master's current role is: FOLLOWER
I20250901 20:02:15.710855 21897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: configured and running, proceeding with master startup.
W20250901 20:02:15.715528 21914 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250901 20:02:15.715600 21914 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250901 20:02:15.717519 21829 master_runner.cc:428] Detected that this master 5f0800e54af54372b1e66ac2c3f428ea is joining an existing cluster
I20250901 20:02:15.717571 21829 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.67.125:45013
I20250901 20:02:15.726608 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.719492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:35918)
0901 20:02:15.719885 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:15.719901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:15.720004 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:15.720399 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:15.720402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:15.720625 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:15.720946 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.720954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.721706 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.721708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:15.722213 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.722221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.722330 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.723025 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:15.723036 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:15.723313 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:15.725308 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.725310 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.725312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.725509 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.725766 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.725769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.725770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.725827 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.726179 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:15.726181 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:15.726248 (+ 67us) client_negotiation.cc:769] Sending connection context
0901 20:02:15.726326 (+ 78us) client_negotiation.cc:240] Negotiation successful
0901 20:02:15.726419 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":311,"threads_started":1}
I20250901 20:02:15.726696 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.719552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.14.67.126:39725)
0901 20:02:15.719743 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.719747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.719985 (+ 238us) server_negotiation.cc:407] Connection header received
0901 20:02:15.720139 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.720142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.720194 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.720274 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.721074 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.721587 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.722332 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.722564 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.723430 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.723447 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.723451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.723477 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.725205 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.725620 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.725624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.725626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.725677 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.725910 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.725913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.725915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.726095 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.726198 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.726393 (+ 195us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.726531 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756934, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.126@KRBTEST.COM for oryx/127.14.67.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.14.67.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.124@KRBTEST.COM" created.
I20250901 20:02:15.754565 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.730571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.125:45013 (local address 127.0.0.1:53120)
0901 20:02:15.730883 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:15.730918 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:15.731038 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:15.735131 (+ 4093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:15.735137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:15.735475 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:15.735664 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.735681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.736408 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.736412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:15.736744 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.736750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.736850 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.737371 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:15.737387 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:15.738728 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:15.752633 (+ 13905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.752643 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.752649 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.752964 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.753320 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.753326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.753331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.753397 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.753848 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:15.753856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:15.753983 (+ 127us) client_negotiation.cc:769] Sending connection context
0901 20:02:15.754144 (+ 161us) client_negotiation.cc:240] Negotiation successful
0901 20:02:15.754344 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20250901 20:02:15.755277 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.734633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.14.67.125:45013)
0901 20:02:15.734888 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.734891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.734902 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:02:15.734951 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.734953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.734995 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.735077 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.735801 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.736306 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.736842 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.737336 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.738842 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.738855 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.738858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.738878 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.752490 (+ 13612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.753109 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.753115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.753119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.753210 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.753505 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.753511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.753516 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.753724 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.753826 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.754975 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.755094 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
I20250901 20:02:15.756574 21921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.719648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.125:45013 (local address 127.0.0.1:53112)
0901 20:02:15.719985 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:15.719996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:15.720084 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:15.720316 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:15.720319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:15.720563 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:15.721013 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.721020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.721902 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.721905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:15.722376 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.722385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.722836 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.751364 (+ 28528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:15.751385 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:15.751809 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:15.755112 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.755115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.755117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.755308 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.755549 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.755554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.755559 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.755613 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.755971 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:15.755977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:15.756064 (+ 87us) client_negotiation.cc:769] Sending connection context
0901 20:02:15.756167 (+ 103us) client_negotiation.cc:240] Negotiation successful
0901 20:02:15.756298 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":85,"threads_started":1}
I20250901 20:02:15.757562 21355 catalog_manager.cc:7069] Initiating ChangeConfig request to add master 127.14.67.125:45013: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea"
member_type: NON_VOTER
last_known_addr {
host: "127.14.67.125"
port: 45013
}
attrs {
promote: true
}
}
dest_uuid: "92c3ebb531304adab92f7135e8afbb43"
cas_config_opid_index: -1
I20250901 20:02:15.757740 21901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.719543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.14.67.125:45013)
0901 20:02:15.719640 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.719645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.720061 (+ 416us) server_negotiation.cc:407] Connection header received
0901 20:02:15.720201 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.720203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.720233 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.720304 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.721121 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.721796 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.722547 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.722733 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.752007 (+ 29274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.752025 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.752030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.752058 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.755012 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.755412 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.755418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.755422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.755465 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.755695 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.755700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.755705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.755859 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.755938 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.757414 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.757542 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20250901 20:02:15.758181 21355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true } }
I20250901 20:02:15.760241 21927 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5f0800e54af54372b1e66ac2c3f428ea (127.14.67.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true } } }
I20250901 20:02:15.760793 21871 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:15.762143 21871 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 FOLLOWER]: Refusing update from remote peer 92c3ebb531304adab92f7135e8afbb43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250901 20:02:15.762183 21927 catalog_manager.cc:7073] Successfully completed master ChangeConfig request to add master 127.14.67.125:45013
I20250901 20:02:15.762753 21829 master.cc:561] Master@127.14.67.125:45013 shutting down...
I20250901 20:02:15.762784 21926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true } } }
I20250901 20:02:15.762864 21926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:15.762859 21927 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 20:02:15.765292 21343 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20250901 20:02:15.766091 21343 consensus_peers.cc:489] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 -> Peer 5f0800e54af54372b1e66ac2c3f428ea (127.14.67.125:45013): Couldn't send request to peer 5f0800e54af54372b1e66ac2c3f428ea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.67.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
I20250901 20:02:15.768800 21829 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:15.769038 21829 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:15.769130 21829 tablet_replica.cc:331] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: stopping tablet replica
I20250901 20:02:15.773814 21829 master.cc:583] Master@127.14.67.125:45013 shutdown complete.
I20250901 20:02:15.773865 21829 master_runner.cc:305] Clearing existing system tablet
I20250901 20:02:15.774083 21829 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:15.774137 21829 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:15.775419 21829 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
WARNING: no policy specified for HTTP/127.14.67.124@KRBTEST.COM; defaulting to no policy
I20250901 20:02:15.776335 21932 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:15.776568 21829 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:15.776690 21829 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250901 20:02:15.776722 21829 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/wal
uuid: "5f0800e54af54372b1e66ac2c3f428ea"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "327969d131fae26fea1dfb6f690ab8c8"
server_key_iv: "54cbdb21b1a14997c767f45339e24e8c"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.124@KRBTEST.COM" created.
I20250901 20:02:15.778968 21829 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:15.781071 21829 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20250901 20:02:15.781153 21829 log.cc:1199] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20250901 20:02:15.781394 21829 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: Deleting consensus metadata
I20250901 20:02:15.781550 21829 master_runner.cc:315] Copying system tablet from 127.14.67.126:39725
I20250901 20:02:15.782660 21829 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: tablet copy: Beginning tablet copy session from remote peer at address 127.14.67.126:39725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.67.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab.
I20250901 20:02:15.791384 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.124:45819
--webserver_interface=127.14.67.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:34599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:39725,127.14.67.125:45013,127.14.67.124:45819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250901 20:02:15.792667 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.783240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.14.67.126:39725)
0901 20:02:15.783364 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.783368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.783607 (+ 239us) server_negotiation.cc:407] Connection header received
0901 20:02:15.783722 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.783728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.783794 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.783892 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.784513 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.785087 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.785687 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.785878 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.786739 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.786758 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.786765 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.786796 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.790924 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.791402 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.791407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.791409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.791461 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.791778 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.791782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.791785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.791978 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.792087 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.792443 (+ 356us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.792555 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250901 20:02:15.793057 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.783146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:35926)
0901 20:02:15.783503 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:15.783516 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:15.783623 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:15.783911 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:15.783914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:15.784110 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:15.784384 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.784392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.785204 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.785207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:15.785583 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.785590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.785694 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.786308 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:15.786324 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:15.786626 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:15.791061 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.791065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.791067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.791296 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.791579 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.791586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.791590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.791657 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.792080 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:15.792087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:15.792227 (+ 140us) client_negotiation.cc:769] Sending connection context
0901 20:02:15.792797 (+ 570us) client_negotiation.cc:240] Negotiation successful
0901 20:02:15.792933 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":80,"threads_started":1}
I20250901 20:02:15.793606 21377 tablet_copy_service.cc:140] P 92c3ebb531304adab92f7135e8afbb43: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5f0800e54af54372b1e66ac2c3f428ea ({username='oryx', principal='oryx/127.14.67.125@KRBTEST.COM'} at 127.0.0.1:35926)
I20250901 20:02:15.793761 21377 tablet_copy_service.cc:161] P 92c3ebb531304adab92f7135e8afbb43: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5f0800e54af54372b1e66ac2c3f428ea at {username='oryx', principal='oryx/127.14.67.125@KRBTEST.COM'} at 127.0.0.1:35926: session id = 5f0800e54af54372b1e66ac2c3f428ea-00000000000000000000000000000000
I20250901 20:02:15.795272 21377 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43: Tablet Copy: opened 0 blocks and 1 log segments
I20250901 20:02:15.796810 21829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20250901 20:02:15.799469 21829 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: tablet copy: Starting download of 0 data blocks...
I20250901 20:02:15.799657 21829 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: tablet copy: Starting download of 1 WAL segments...
I20250901 20:02:15.801564 21829 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250901 20:02:15.803498 21377 tablet_copy_service.cc:342] P 92c3ebb531304adab92f7135e8afbb43: Request end of tablet copy session 5f0800e54af54372b1e66ac2c3f428ea-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.67.125@KRBTEST.COM'} at 127.0.0.1:35926
I20250901 20:02:15.803610 21377 tablet_copy_service.cc:434] P 92c3ebb531304adab92f7135e8afbb43: ending tablet copy session 5f0800e54af54372b1e66ac2c3f428ea-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5f0800e54af54372b1e66ac2c3f428ea
W20250901 20:02:15.808117 21829 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250901 20:02:15.810277 21829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:15.812002 21941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:15.812018 21944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:15.812299 21829 server_base.cc:1047] running on GCE node
W20250901 20:02:15.812427 21942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:15.812613 21829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:15.812870 21829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:15.814006 21829 hybrid_clock.cc:648] HybridClock initialized: now 1756756935813995 us; error 29 us; skew 500 ppm
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:15.815722 21829 init.cc:377] Logged in from keytab as oryx/127.14.67.125@KRBTEST.COM (short username oryx)
I20250901 20:02:15.816169 21829 webserver.cc:480] Webserver started at http://127.14.67.125:36895/ using document root <none> and password file <none>
I20250901 20:02:15.816354 21829 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:15.816397 21829 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:15.817571 21829 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20250901 20:02:15.818611 21951 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:15.818873 21829 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250901 20:02:15.818948 21829 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/wal
uuid: "5f0800e54af54372b1e66ac2c3f428ea"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "327969d131fae26fea1dfb6f690ab8c8"
server_key_iv: "54cbdb21b1a14997c767f45339e24e8c"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.819025 21829 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:15.828362 21829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:15.830307 21829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:15.834852 21829 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.125:45013
I20250901 20:02:15.834882 22003 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.125:45013 every 8 connection(s)
I20250901 20:02:15.835407 21829 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-1/data/info.pb
I20250901 20:02:15.837452 22004 sys_catalog.cc:263] Verifying existing consensus state
I20250901 20:02:15.838413 22004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: Bootstrap starting.
I20250901 20:02:15.847834 22004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 20:02:15.848136 22004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: Bootstrap complete.
I20250901 20:02:15.848692 22004 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true } }
I20250901 20:02:15.848810 22004 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f0800e54af54372b1e66ac2c3f428ea, State: Initialized, Role: LEARNER
I20250901 20:02:15.848912 22004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true } }
I20250901 20:02:15.849282 22007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true } } }
I20250901 20:02:15.849356 22004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:15.849449 22007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: This master's current role is: LEARNER
I20250901 20:02:15.852187 22018 catalog_manager.cc:1261] Loaded cluster ID: edd92a8d72a04b938d695a74fda08800
I20250901 20:02:15.852296 22018 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: loading cluster ID for follower catalog manager: success
I20250901 20:02:15.854410 22018 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: acquiring CA information for follower catalog manager: success
I20250901 20:02:15.855746 22018 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250901 20:02:15.902331 21940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:15.902578 21940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:15.902688 21940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:15.905683 21940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:15.905733 21940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:15.905756 21940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:15.905776 21940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:15.905809 21940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:02:15.909713 21940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:34599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:39725,127.14.67.125:45013,127.14.67.124:45819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.124:45819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.67.124
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:15.910694 21940 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:15.911573 21940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:15.917172 22023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:15.917273 22026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:15.917338 22024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:15.917727 21940 server_base.cc:1047] running on GCE node
I20250901 20:02:15.918162 21940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:15.919027 21940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:15.920182 21940 hybrid_clock.cc:648] HybridClock initialized: now 1756756935920168 us; error 34 us; skew 500 ppm
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:15.922665 21940 init.cc:377] Logged in from keytab as oryx/127.14.67.124@KRBTEST.COM (short username oryx)
I20250901 20:02:15.923586 21940 webserver.cc:480] Webserver started at http://127.14.67.124:37785/ using document root <none> and password file <none>
I20250901 20:02:15.924084 21940 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:15.924130 21940 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:15.924290 21940 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:15.925905 21940 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data/instance:
uuid: "1c0d1c7626a0438b866868946ac054e2"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "e8b3d6e710a971dcd7b76d9c06ed8777"
server_key_iv: "a60a308613ee2975f8f744ff35713cbf"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.926395 21940 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/wal/instance:
uuid: "1c0d1c7626a0438b866868946ac054e2"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "e8b3d6e710a971dcd7b76d9c06ed8777"
server_key_iv: "a60a308613ee2975f8f744ff35713cbf"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.929435 21940 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250901 20:02:15.931391 22033 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:15.932281 21940 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:15.932382 21940 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/wal
uuid: "1c0d1c7626a0438b866868946ac054e2"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "e8b3d6e710a971dcd7b76d9c06ed8777"
server_key_iv: "a60a308613ee2975f8f744ff35713cbf"
server_key_version: "encryptionkey@0"
I20250901 20:02:15.932466 21940 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:15.948284 21940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:15.951678 21940 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:15.951859 21940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:15.958796 21940 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.124:45819
I20250901 20:02:15.958799 22085 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.124:45819 every 8 connection(s)
I20250901 20:02:15.959734 21940 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data/info.pb
I20250901 20:02:15.961973 22086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:15.965541 22086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } has no permanent_uuid. Determining permanent_uuid...
I20250901 20:02:15.966980 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 21940
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.124@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.67.124@KRBTEST.COM
I20250901 20:02:15.979213 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.967164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.14.67.126:39725)
0901 20:02:15.967440 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.967446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.967898 (+ 452us) server_negotiation.cc:407] Connection header received
0901 20:02:15.968792 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.968796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.968866 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.968983 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.970277 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.971191 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.972151 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.972391 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.975077 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.975096 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.975098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.975121 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.976938 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.977423 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.977430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.977433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.977484 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.978045 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.978048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.978050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.978236 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.978356 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.978932 (+ 576us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.979053 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20250901 20:02:15.979847 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.967336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:35930)
0901 20:02:15.967762 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:15.967803 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:15.968596 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:15.969076 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:15.969085 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:15.969602 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:15.970100 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.970118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.971356 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.971360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:15.971982 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.971992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.972232 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.973144 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:15.973187 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:15.974924 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:15.977054 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.977062 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.977069 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.977325 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.977658 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.977663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.977669 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.977869 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.978398 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:15.978405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:15.978696 (+ 291us) client_negotiation.cc:769] Sending connection context
0901 20:02:15.978876 (+ 180us) client_negotiation.cc:240] Negotiation successful
0901 20:02:15.979155 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":97,"threads_started":1}
I20250901 20:02:15.979852 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.968107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.14.67.124:45819)
0901 20:02:15.968483 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.968488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.968527 (+ 39us) server_negotiation.cc:407] Connection header received
0901 20:02:15.968695 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.968719 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.969001 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.969188 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.970250 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.971408 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.972353 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.972624 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.975538 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.975559 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.975566 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.975589 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.977583 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.978172 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.978177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.978179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.978232 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.978613 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.978616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.978617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.978788 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.978975 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.979171 (+ 196us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.979304 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":161,"threads_started":1}
I20250901 20:02:15.980952 22086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } has no permanent_uuid. Determining permanent_uuid...
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.124@KRBTEST.COM for oryx/127.14.67.125@KRBTEST.COM
I20250901 20:02:15.988910 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.981347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.125:45013 (local address 127.0.0.1:53128)
0901 20:02:15.981460 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:15.981475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:15.981539 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:15.981902 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:15.981904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:15.982034 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:15.982179 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.982184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.983127 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.983135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:15.983548 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.983556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.983672 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.984353 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:15.984367 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:15.985720 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:15.987633 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.987636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.987638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.987816 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.988061 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.988063 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.988065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.988110 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.988434 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:15.988437 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:15.988525 (+ 88us) client_negotiation.cc:769] Sending connection context
0901 20:02:15.988627 (+ 102us) client_negotiation.cc:240] Negotiation successful
0901 20:02:15.988742 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20250901 20:02:15.989048 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.981476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.14.67.125:45013)
0901 20:02:15.981716 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.981719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.981730 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:02:15.981763 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.981765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.981802 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.981882 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.982278 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.982977 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.983716 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.983941 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.985870 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.985886 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.985889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.985913 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.987506 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.987937 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.987939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.987940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.987975 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.988184 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.988186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.988187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.988315 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.988390 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.988739 (+ 349us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.988889 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20250901 20:02:15.989956 22086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } has no permanent_uuid. Determining permanent_uuid...
Sep 01 20:02:15 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.124@KRBTEST.COM for oryx/127.14.67.124@KRBTEST.COM
I20250901 20:02:15.997035 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.990460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.124:45819 (local address 127.0.0.1:47534)
0901 20:02:15.990678 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:15.990696 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:15.990776 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:15.990985 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:15.990988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:15.991142 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:15.991287 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.991292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.991924 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.991929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:15.992284 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:15.992290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.992387 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.992899 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:15.992912 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:15.994203 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:15.995972 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.995975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.995976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.996133 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.996343 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:15.996346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:15.996347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:15.996389 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:15.996662 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:15.996664 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:15.996728 (+ 64us) client_negotiation.cc:769] Sending connection context
0901 20:02:15.996810 (+ 82us) client_negotiation.cc:240] Negotiation successful
0901 20:02:15.996892 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20250901 20:02:15.997061 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:15.990567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.14.67.124:45819)
0901 20:02:15.990752 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:15.990756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:15.990768 (+ 12us) server_negotiation.cc:407] Connection header received
0901 20:02:15.990865 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:15.990868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:15.990905 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:15.990970 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:15.991388 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.991818 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:15.992381 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:15.992581 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:15.994353 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:15.994365 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:15.994367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:15.994387 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:15.995865 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.996209 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.996212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.996213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.996253 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:15.996454 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:15.996456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:15.996457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:15.996563 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:15.996629 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:15.996848 (+ 219us) server_negotiation.cc:299] Negotiation successful
0901 20:02:15.996940 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":92}
I20250901 20:02:15.999615 22086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: Bootstrap starting.
I20250901 20:02:16.001969 22086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:16.002718 22086 log.cc:826] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:16.004592 22086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: No bootstrap required, opened a new log
I20250901 20:02:16.006922 22086 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } }
I20250901 20:02:16.007073 22086 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:16.007107 22086 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0d1c7626a0438b866868946ac054e2, State: Initialized, Role: FOLLOWER
I20250901 20:02:16.007508 22086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } }
I20250901 20:02:16.008038 22092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } } }
I20250901 20:02:16.008224 22092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 20:02:16.008491 22086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: configured and running, proceeding with master startup.
W20250901 20:02:16.012907 22103 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250901 20:02:16.012997 22103 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250901 20:02:16.016589 21940 master_runner.cc:428] Detected that this master 1c0d1c7626a0438b866868946ac054e2 is joining an existing cluster
I20250901 20:02:16.016650 21940 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.67.124:45819
I20250901 20:02:16.025434 21550 heartbeater.cc:499] Master 127.14.67.126:39725 was elected leader, sending a full tablet report...
I20250901 20:02:16.036645 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.033844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:35938)
0901 20:02:16.033963 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:16.033974 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:16.034040 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:16.034380 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:16.034383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:16.034398 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:16.034647 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.034653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.035377 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.035380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:16.036151 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.036162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.036288 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.036352 (+ 64us) client_negotiation.cc:769] Sending connection context
0901 20:02:16.036410 (+ 58us) client_negotiation.cc:240] Negotiation successful
0901 20:02:16.036477 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"spinlock_wait_cycles":2304}
I20250901 20:02:16.037124 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.033900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.14.67.126:39725)
0901 20:02:16.034206 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:16.034209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:16.034218 (+ 9us) server_negotiation.cc:407] Connection header received
0901 20:02:16.034254 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:16.034256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:16.034289 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:16.034364 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:16.034749 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.035271 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.036269 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.036902 (+ 633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.036950 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:16.036988 (+ 38us) server_negotiation.cc:299] Negotiation successful
0901 20:02:16.037045 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":129,"threads_started":1}
I20250901 20:02:16.041900 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.038176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.125:45013 (local address 127.0.0.1:53152)
0901 20:02:16.038305 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:16.038320 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:16.038396 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:16.038826 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:16.038830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:16.038848 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:16.039060 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.039066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.040188 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.040192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:16.041407 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.041428 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.041569 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.041595 (+ 26us) client_negotiation.cc:769] Sending connection context
0901 20:02:16.041644 (+ 49us) client_negotiation.cc:240] Negotiation successful
0901 20:02:16.041732 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20250901 20:02:16.043653 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.038274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.14.67.125:45013)
0901 20:02:16.038557 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:16.038562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:16.038576 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:02:16.038623 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:16.038626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:16.038682 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:16.039164 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:16.039260 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.040059 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.042482 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.043131 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.043329 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:16.043397 (+ 68us) server_negotiation.cc:299] Negotiation successful
0901 20:02:16.043474 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":94,"threads_started":1}
I20250901 20:02:16.047994 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.019437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47542 (local address 127.14.67.124:45819)
0901 20:02:16.019557 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:16.019561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:16.020223 (+ 662us) server_negotiation.cc:407] Connection header received
0901 20:02:16.020330 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:16.020334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:16.020381 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:16.020460 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:16.039629 (+ 19169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.040236 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.041374 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.041584 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.043684 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:16.043701 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:16.043704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:16.043729 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:16.046357 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:16.046848 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:16.046851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:16.046853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:16.046894 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:16.047122 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:16.047126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:16.047128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:16.047314 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:16.047392 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:16.047678 (+ 286us) server_negotiation.cc:299] Negotiation successful
0901 20:02:16.047803 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250901 20:02:16.048522 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.018831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.125:45013 (local address 127.0.0.1:53140)
0901 20:02:16.019347 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:16.019363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:16.019493 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:16.019830 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:16.019833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:16.038843 (+ 19010us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:16.039200 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.039213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.042584 (+ 3371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.042588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:16.043213 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.043223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.043362 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.044051 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:16.044064 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:16.044496 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:16.047061 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:16.047066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:16.047069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:16.047255 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:16.047510 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:16.047513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:16.047516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:16.047557 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:16.047978 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:16.047981 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:16.048062 (+ 81us) client_negotiation.cc:769] Sending connection context
0901 20:02:16.048183 (+ 121us) client_negotiation.cc:240] Negotiation successful
0901 20:02:16.048335 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"spinlock_wait_cycles":9344,"thread_start_us":103,"threads_started":1}
I20250901 20:02:16.048581 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.019835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.124:45819 (local address 127.0.0.1:47542)
0901 20:02:16.020162 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:16.020174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:16.020249 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:16.020504 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:16.020507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:16.038837 (+ 18330us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:16.039523 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.039531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.040382 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.040386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:16.041132 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.041156 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.041750 (+ 594us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.042948 (+ 1198us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:16.042971 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:16.043560 (+ 589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:16.046497 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:16.046527 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:16.046530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:16.046742 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:16.046980 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:16.046983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:16.046984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:16.047030 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:16.047432 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:16.047436 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:16.047523 (+ 87us) client_negotiation.cc:769] Sending connection context
0901 20:02:16.048226 (+ 703us) client_negotiation.cc:240] Negotiation successful
0901 20:02:16.048422 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":89,"spinlock_wait_cycles":2176,"thread_start_us":99,"threads_started":1}
I20250901 20:02:16.049010 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.019053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:35934)
0901 20:02:16.020057 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:16.020101 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:16.020196 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:16.020541 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:16.020546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:16.040225 (+ 19679us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:16.040507 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.040513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.041628 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.041632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:16.042200 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.042209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.042339 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.044266 (+ 1927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:16.044287 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:16.044750 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:16.047299 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:16.047303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:16.047305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:16.047522 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:16.047768 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:16.047771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:16.047773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:16.047821 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:16.048520 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:16.048524 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:16.048601 (+ 77us) client_negotiation.cc:769] Sending connection context
0901 20:02:16.048702 (+ 101us) client_negotiation.cc:240] Negotiation successful
0901 20:02:16.048828 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"spinlock_wait_cycles":5504,"thread_start_us":115,"threads_started":1}
I20250901 20:02:16.049396 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.018899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.14.67.125:45013)
0901 20:02:16.019178 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:16.019185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:16.019430 (+ 245us) server_negotiation.cc:407] Connection header received
0901 20:02:16.019600 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:16.019604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:16.019650 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:16.019699 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:16.039379 (+ 19680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.040112 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.043584 (+ 3472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.043828 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.044626 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:16.044642 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:16.044647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:16.044669 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:16.046742 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:16.047368 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:16.047372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:16.047373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:16.047420 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:16.047652 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:16.047655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:16.047657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:16.047838 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:16.047937 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:16.049080 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0901 20:02:16.049242 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20250901 20:02:16.050174 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.018834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.14.67.126:39725)
0901 20:02:16.018972 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:16.018977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:16.020174 (+ 1197us) server_negotiation.cc:407] Connection header received
0901 20:02:16.020320 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:16.020324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:16.020375 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:16.020415 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:16.040620 (+ 20205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.041463 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.042542 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.042747 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.044921 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:16.044942 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:16.044948 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:16.044984 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:16.047169 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:16.047628 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:16.047632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:16.047634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:16.047679 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:16.047924 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:16.047927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:16.047930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:16.048117 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:16.048205 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:16.049895 (+ 1690us) server_negotiation.cc:299] Negotiation successful
0901 20:02:16.050017 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 01 20:02:16 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756934, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.126@KRBTEST.COM for oryx/127.14.67.124@KRBTEST.COM
I20250901 20:02:16.063148 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.054393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.124:45819 (local address 127.0.0.1:47556)
0901 20:02:16.054700 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:16.054721 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:16.054828 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:16.055428 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:16.055431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:16.055606 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:16.055800 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.055810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.056641 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.056647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:16.057075 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.057084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.057197 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.057768 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:16.057785 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:16.059543 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:16.061598 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:16.061602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:16.061604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:16.061775 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:16.061989 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:16.061994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:16.061996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:16.062050 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:16.062387 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:16.062390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:16.062496 (+ 106us) client_negotiation.cc:769] Sending connection context
0901 20:02:16.062630 (+ 134us) client_negotiation.cc:240] Negotiation successful
0901 20:02:16.062775 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20250901 20:02:16.063454 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.054484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.14.67.124:45819)
0901 20:02:16.054723 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:16.054727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:16.054774 (+ 47us) server_negotiation.cc:407] Connection header received
0901 20:02:16.054969 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:16.054971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:16.055049 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:16.055275 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:16.055935 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.056548 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.057424 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.057600 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.059682 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:16.059702 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:16.059705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:16.059732 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:16.061477 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:16.061864 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:16.061870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:16.061872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:16.061909 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:16.062130 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:16.062133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:16.062135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:16.062297 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:16.062386 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:16.062681 (+ 295us) server_negotiation.cc:299] Negotiation successful
0901 20:02:16.062966 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20250901 20:02:16.064368 21355 catalog_manager.cc:7069] Initiating ChangeConfig request to add master 127.14.67.124:45819: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1c0d1c7626a0438b866868946ac054e2"
member_type: NON_VOTER
last_known_addr {
host: "127.14.67.124"
port: 45819
}
attrs {
promote: true
}
}
dest_uuid: "92c3ebb531304adab92f7135e8afbb43"
cas_config_opid_index: 5
I20250901 20:02:16.064903 21355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } }
I20250901 20:02:16.065956 21927 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1c0d1c7626a0438b866868946ac054e2 (127.14.67.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:16.066368 22060 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:16.066725 21978 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 LEARNER]: Refusing update from remote peer 92c3ebb531304adab92f7135e8afbb43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250901 20:02:16.067193 21926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250901 20:02:16.067474 21927 catalog_manager.cc:7073] Successfully completed master ChangeConfig request to add master 127.14.67.124:45819
I20250901 20:02:16.067628 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:16.067725 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:16.068341 21940 master.cc:561] Master@127.14.67.124:45819 shutting down...
I20250901 20:02:16.068733 22060 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 FOLLOWER]: Refusing update from remote peer 92c3ebb531304adab92f7135e8afbb43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250901 20:02:16.068998 21978 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1c0d1c7626a0438b866868946ac054e2 (127.14.67.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:16.069888 21926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:16.071039 22007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92c3ebb531304adab92f7135e8afbb43. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:16.071130 22007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: This master's current role is: LEARNER
I20250901 20:02:16.071211 22007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: NON_VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: true } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:16.071280 22007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: This master's current role is: LEARNER
W20250901 20:02:16.071581 21342 consensus_peers.cc:489] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 -> Peer 1c0d1c7626a0438b866868946ac054e2 (127.14.67.124:45819): Couldn't send request to peer 1c0d1c7626a0438b866868946ac054e2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20250901 20:02:16.073242 21940 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:16.073470 21940 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:16.073565 21940 tablet_replica.cc:331] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: stopping tablet replica
I20250901 20:02:16.074121 21926 raft_consensus.cc:1062] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43: attempting to promote NON_VOTER 5f0800e54af54372b1e66ac2c3f428ea to VOTER
I20250901 20:02:16.074553 21926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } }
I20250901 20:02:16.075665 21978 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 LEARNER]: Refusing update from remote peer 92c3ebb531304adab92f7135e8afbb43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250901 20:02:16.076296 22116 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250901 20:02:16.076519 21342 consensus_peers.cc:489] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 -> Peer 1c0d1c7626a0438b866868946ac054e2 (127.14.67.124:45819): Couldn't send request to peer 1c0d1c7626a0438b866868946ac054e2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20250901 20:02:16.078790 21927 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5f0800e54af54372b1e66ac2c3f428ea (127.14.67.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:16.079392 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.079126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.124:45819 (local address 127.0.0.1:47560)
0901 20:02:16.079257 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:16.079318 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.124:45819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20250901 20:02:16.079579 21355 master.cc:679] Network error: unable to get registration information for peer 1c0d1c7626a0438b866868946ac054e2: Client connection negotiation failed: client connection to 127.14.67.124:45819: connect: Connection refused (error 111)
I20250901 20:02:16.080142 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:16.080238 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:16.080391 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:16.080466 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:16.080811 21940 master.cc:583] Master@127.14.67.124:45819 shutdown complete.
I20250901 20:02:16.080854 21940 master_runner.cc:305] Clearing existing system tablet
I20250901 20:02:16.080848 21978 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5f0800e54af54372b1e66ac2c3f428ea (127.14.67.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:16.081097 21940 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:16.081144 21940 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:16.082468 21940 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250901 20:02:16.082466 22115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:16.082713 22115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: This master's current role is: FOLLOWER
I20250901 20:02:16.083387 22118 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:16.083639 21940 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:16.083765 21940 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250901 20:02:16.083798 21940 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/wal
uuid: "1c0d1c7626a0438b866868946ac054e2"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "e8b3d6e710a971dcd7b76d9c06ed8777"
server_key_iv: "a60a308613ee2975f8f744ff35713cbf"
server_key_version: "encryptionkey@0"
I20250901 20:02:16.085546 21940 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:16.087625 21940 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20250901 20:02:16.087706 21940 log.cc:1199] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20250901 20:02:16.087935 21940 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: Deleting consensus metadata
I20250901 20:02:16.088088 21940 master_runner.cc:315] Copying system tablet from 127.14.67.126:39725
I20250901 20:02:16.089171 21940 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: tablet copy: Beginning tablet copy session from remote peer at address 127.14.67.126:39725
I20250901 20:02:16.096655 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.089732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:35940)
0901 20:02:16.090051 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:16.090068 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:16.090196 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:16.090492 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:16.090496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:16.090724 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:16.090911 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.090917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.091928 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.091934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:16.092331 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.092339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.092430 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.093195 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:16.093211 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:16.093563 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:16.095531 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:16.095535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:16.095536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:16.095705 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:16.095917 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:16.095919 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:16.095921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:16.095959 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:16.096286 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:16.096289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:16.096352 (+ 63us) client_negotiation.cc:769] Sending connection context
0901 20:02:16.096432 (+ 80us) client_negotiation.cc:240] Negotiation successful
0901 20:02:16.096531 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":129,"threads_started":1}
I20250901 20:02:16.096730 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.089812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.14.67.126:39725)
0901 20:02:16.089929 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:16.089933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:16.090162 (+ 229us) server_negotiation.cc:407] Connection header received
0901 20:02:16.090286 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:16.090290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:16.090338 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:16.090436 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:16.091026 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.091806 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.092431 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.092621 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.093687 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:16.093703 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:16.093706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:16.093731 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:16.095417 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:16.095793 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:16.095796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:16.095798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:16.095842 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:16.096029 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:16.096032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:16.096034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:16.096192 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:16.096299 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:16.096491 (+ 192us) server_negotiation.cc:299] Negotiation successful
0901 20:02:16.096592 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250901 20:02:16.097277 21377 tablet_copy_service.cc:140] P 92c3ebb531304adab92f7135e8afbb43: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1c0d1c7626a0438b866868946ac054e2 ({username='oryx', principal='oryx/127.14.67.124@KRBTEST.COM'} at 127.0.0.1:35940)
I20250901 20:02:16.097362 21377 tablet_copy_service.cc:161] P 92c3ebb531304adab92f7135e8afbb43: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1c0d1c7626a0438b866868946ac054e2 at {username='oryx', principal='oryx/127.14.67.124@KRBTEST.COM'} at 127.0.0.1:35940: session id = 1c0d1c7626a0438b866868946ac054e2-00000000000000000000000000000000
I20250901 20:02:16.098197 21377 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43: Tablet Copy: opened 0 blocks and 1 log segments
I20250901 20:02:16.099277 21940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20250901 20:02:16.099325 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.099094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.124:45819 (local address 127.0.0.1:47564)
0901 20:02:16.099208 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:16.099265 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.124:45819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20250901 20:02:16.099507 21355 master.cc:679] Network error: unable to get registration information for peer 1c0d1c7626a0438b866868946ac054e2: Client connection negotiation failed: client connection to 127.14.67.124:45819: connect: Connection refused (error 111)
I20250901 20:02:16.100809 21940 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: tablet copy: Starting download of 0 data blocks...
I20250901 20:02:16.100960 21940 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: tablet copy: Starting download of 1 WAL segments...
I20250901 20:02:16.102206 21940 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250901 20:02:16.103662 21377 tablet_copy_service.cc:342] P 92c3ebb531304adab92f7135e8afbb43: Request end of tablet copy session 1c0d1c7626a0438b866868946ac054e2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.67.124@KRBTEST.COM'} at 127.0.0.1:35940
I20250901 20:02:16.103806 21377 tablet_copy_service.cc:434] P 92c3ebb531304adab92f7135e8afbb43: ending tablet copy session 1c0d1c7626a0438b866868946ac054e2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1c0d1c7626a0438b866868946ac054e2
W20250901 20:02:16.109184 21940 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250901 20:02:16.111752 21940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:16.113575 22125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:16.113590 22128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:16.113864 22126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:16.114701 21940 server_base.cc:1047] running on GCE node
I20250901 20:02:16.114879 21940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:16.115156 21940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:16.116307 21940 hybrid_clock.cc:648] HybridClock initialized: now 1756756936116294 us; error 31 us; skew 500 ppm
Sep 01 20:02:16 dist-test-slave-2s4d krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:16.117796 21940 init.cc:377] Logged in from keytab as oryx/127.14.67.124@KRBTEST.COM (short username oryx)
I20250901 20:02:16.118242 21940 webserver.cc:480] Webserver started at http://127.14.67.124:46855/ using document root <none> and password file <none>
I20250901 20:02:16.118420 21940 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:16.118455 21940 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:16.119477 21940 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250901 20:02:16.120445 22135 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:16.120667 21940 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250901 20:02:16.120731 21940 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/wal
uuid: "1c0d1c7626a0438b866868946ac054e2"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "e8b3d6e710a971dcd7b76d9c06ed8777"
server_key_iv: "a60a308613ee2975f8f744ff35713cbf"
server_key_version: "encryptionkey@0"
I20250901 20:02:16.120806 21940 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:16.134855 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.134616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.124:45819 (local address 127.0.0.1:47574)
0901 20:02:16.134733 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:16.134783 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.124:45819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20250901 20:02:16.135036 21355 master.cc:679] Network error: unable to get registration information for peer 1c0d1c7626a0438b866868946ac054e2: Client connection negotiation failed: client connection to 127.14.67.124:45819: connect: Connection refused (error 111)
I20250901 20:02:16.141616 21940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:16.143326 21940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:16.146920 21940 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.124:45819
I20250901 20:02:16.147001 22187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.124:45819 every 8 connection(s)
I20250901 20:02:16.147419 21940 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/master-2/data/info.pb
I20250901 20:02:16.148715 22188 sys_catalog.cc:263] Verifying existing consensus state
I20250901 20:02:16.149423 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: Bootstrap starting.
I20250901 20:02:16.157670 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 20:02:16.157910 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: Bootstrap complete.
I20250901 20:02:16.158391 22188 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } }
I20250901 20:02:16.158488 22188 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c0d1c7626a0438b866868946ac054e2, State: Initialized, Role: LEARNER
I20250901 20:02:16.158620 22188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } }
I20250901 20:02:16.158922 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:16.159011 22188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:16.159040 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: This master's current role is: LEARNER
I20250901 20:02:16.162338 22202 catalog_manager.cc:1261] Loaded cluster ID: edd92a8d72a04b938d695a74fda08800
I20250901 20:02:16.162447 22202 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: loading cluster ID for follower catalog manager: success
I20250901 20:02:16.164239 22202 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: acquiring CA information for follower catalog manager: success
I20250901 20:02:16.165274 22202 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 20:02:16.205823 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.203098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.124:45819 (local address 127.0.0.1:47586)
0901 20:02:16.203268 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:16.203281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:16.203348 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:16.203622 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:16.203625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:16.203641 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:16.203810 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.203816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.204726 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.204730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:16.205441 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:16.205450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.205557 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.205588 (+ 31us) client_negotiation.cc:769] Sending connection context
0901 20:02:16.205635 (+ 47us) client_negotiation.cc:240] Negotiation successful
0901 20:02:16.205694 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20250901 20:02:16.206233 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:16.203197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.14.67.124:45819)
0901 20:02:16.203409 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:16.203412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:16.203424 (+ 12us) server_negotiation.cc:407] Connection header received
0901 20:02:16.203464 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:16.203466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:16.203511 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:16.203594 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:16.203909 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.204567 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:16.205573 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:16.205891 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:16.205996 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:16.206054 (+ 58us) server_negotiation.cc:299] Negotiation successful
0901 20:02:16.206122 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":76,"threads_started":1}
I20250901 20:02:16.251420 21686 heartbeater.cc:499] Master 127.14.67.126:39725 was elected leader, sending a full tablet report...
I20250901 20:02:16.460716 21822 heartbeater.cc:499] Master 127.14.67.126:39725 was elected leader, sending a full tablet report...
I20250901 20:02:16.643551 22162 raft_consensus.cc:1215] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250901 20:02:16.643922 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92c3ebb531304adab92f7135e8afbb43. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:16.644059 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: This master's current role is: LEARNER
I20250901 20:02:17.032174 22204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: NON_VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: true } } }
I20250901 20:02:17.032341 22204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:17.033084 22204 raft_consensus.cc:1062] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43: attempting to promote NON_VOTER 1c0d1c7626a0438b866868946ac054e2 to VOTER
I20250901 20:02:17.033428 22204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } }
I20250901 20:02:17.034436 22162 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 LEARNER]: Refusing update from remote peer 92c3ebb531304adab92f7135e8afbb43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250901 20:02:17.034526 21978 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 FOLLOWER]: Refusing update from remote peer 92c3ebb531304adab92f7135e8afbb43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250901 20:02:17.034904 22206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250901 20:02:17.035110 22204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250901 20:02:17.036684 22206 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c0d1c7626a0438b866868946ac054e2 (127.14.67.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } } }
I20250901 20:02:17.037142 21978 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c0d1c7626a0438b866868946ac054e2 (127.14.67.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } } }
I20250901 20:02:17.038118 22205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } } }
I20250901 20:02:17.038134 22206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } } }
I20250901 20:02:17.038244 22205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:17.038256 22206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:17.038641 22209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } } }
I20250901 20:02:17.038749 22209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: This master's current role is: FOLLOWER
I20250901 20:02:17.038897 22206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } } }
I20250901 20:02:17.038992 22206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:17.039254 22162 raft_consensus.cc:2953] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c0d1c7626a0438b866868946ac054e2 (127.14.67.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } } }
I20250901 20:02:17.040454 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92c3ebb531304adab92f7135e8afbb43" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } } }
I20250901 20:02:17.040552 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 20:02:17.125309 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.118709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.14.67.125:45013)
0901 20:02:17.118941 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.118945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.118960 (+ 15us) server_negotiation.cc:407] Connection header received
0901 20:02:17.118998 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.119001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.119048 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.119142 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.119925 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.120440 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.121090 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.121228 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.122392 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.122408 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.122410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.122431 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.123954 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.124395 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.124398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.124399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.124439 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.124668 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.124671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.124672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.124807 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.124912 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.125090 (+ 178us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.125180 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
I20250901 20:02:17.128571 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.126071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:35952)
0901 20:02:17.126284 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.126296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.126363 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.126579 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.126581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.126594 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:17.126755 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.126759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.127462 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.127464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.128194 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.128203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.128326 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.128347 (+ 21us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.128397 (+ 50us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.128457 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
I20250901 20:02:17.129120 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.126154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.14.67.126:39725)
0901 20:02:17.126368 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.126371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.126384 (+ 13us) server_negotiation.cc:407] Connection header received
0901 20:02:17.126419 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.126421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.126459 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.126571 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:17.126857 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.127366 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.128333 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.128825 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.128868 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.128918 (+ 50us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.128964 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":62,"threads_started":1}
I20250901 20:02:17.132474 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.130191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.125:45013 (local address 127.0.0.1:53170)
0901 20:02:17.130307 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.130319 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.130392 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.130604 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.130607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.130621 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:17.130766 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.130772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.131532 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.131536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.132204 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.132212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.132310 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.132322 (+ 12us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.132362 (+ 40us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.132400 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20250901 20:02:17.132957 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.130291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.14.67.125:45013)
0901 20:02:17.130392 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.130397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.130408 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:02:17.130470 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.130475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.130537 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.130600 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:17.130861 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.131420 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.132331 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.132791 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.132815 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.132851 (+ 36us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.132888 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":1664}
I20250901 20:02:17.136895 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.134103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.124:45819 (local address 127.0.0.1:47588)
0901 20:02:17.134216 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.134232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.134308 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.134708 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.134711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.134729 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:17.134931 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.134941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.135773 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.135779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.136540 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.136548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.136655 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.136676 (+ 21us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.136724 (+ 48us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.136777 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20250901 20:02:17.137615 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.134207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.14.67.124:45819)
0901 20:02:17.134445 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.134449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.134466 (+ 17us) server_negotiation.cc:407] Connection header received
0901 20:02:17.134543 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.134546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.134604 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.134705 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:17.135039 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.135633 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.137049 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.137413 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.137441 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.137481 (+ 40us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.137523 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
I20250901 20:02:17.145720 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.139297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.14.67.124:45819)
0901 20:02:17.139408 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.139411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.139423 (+ 12us) server_negotiation.cc:407] Connection header received
0901 20:02:17.139511 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.139513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.139547 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.139613 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.140720 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.141160 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.141783 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.141898 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.143045 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.143059 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.143061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.143081 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.144526 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.144869 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.144871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.144872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.144905 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.145063 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.145065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.145066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.145179 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.145274 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.145478 (+ 204us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.145562 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250901 20:02:17.148811 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.146385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:35958)
0901 20:02:17.146658 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.146672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.146768 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.147014 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.147017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.147033 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:17.147213 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.147218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.147893 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.147896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.148457 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.148463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.148569 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.148586 (+ 17us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.148635 (+ 49us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.148695 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20250901 20:02:17.149202 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.146540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35958 (local address 127.14.67.126:39725)
0901 20:02:17.146679 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.146685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.146728 (+ 43us) server_negotiation.cc:407] Connection header received
0901 20:02:17.146887 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.146889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.146928 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.146992 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:17.147320 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.147791 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.148571 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.149012 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.149041 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.149081 (+ 40us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.149120 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250901 20:02:17.155033 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.152496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.125:45013 (local address 127.0.0.1:53180)
0901 20:02:17.152676 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.152690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.152753 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.152954 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.152957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.152970 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:17.153107 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.153111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.154039 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.154042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.154672 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.154684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.154799 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.154816 (+ 17us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.154868 (+ 52us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.154918 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20250901 20:02:17.155426 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.152617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53180 (local address 127.14.67.125:45013)
0901 20:02:17.152738 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.152742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.152756 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:02:17.152821 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.152824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.152867 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.152939 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:17.153188 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.153905 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.154792 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.155229 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.155270 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.155308 (+ 38us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.155347 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250901 20:02:17.158834 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.156646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.124:45819 (local address 127.0.0.1:47614)
0901 20:02:17.156756 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.156767 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.156825 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.157058 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.157061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.157077 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:17.157222 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.157227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.157839 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.157843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.158461 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.158469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.158617 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.158633 (+ 16us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.158683 (+ 50us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.158733 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20250901 20:02:17.159183 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.156787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.14.67.124:45819)
0901 20:02:17.156914 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.156920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.156930 (+ 10us) server_negotiation.cc:407] Connection header received
0901 20:02:17.156955 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.156957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.156985 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.157047 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:17.157314 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.157746 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.158602 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.159014 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.159038 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.159082 (+ 44us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.159116 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56,"spinlock_wait_cycles":20608}
I20250901 20:02:17.160370 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 21421
I20250901 20:02:17.166749 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:40197
--local_ip_for_outbound_sockets=127.14.67.65
--tserver_master_addrs=127.14.67.126:39725
--webserver_port=40877
--webserver_interface=127.14.67.65
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:34599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.67.126:39725,127.14.67.125:45013
--tserver_master_addrs=127.14.67.126:39725,127.14.67.125:45013,127.14.67.124:45819 with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:17.271127 22216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:17.271317 22216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:17.271412 22216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:17.274444 22216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:17.274498 22216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:17.274612 22216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:02:17.278414 22216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:34599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:40197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=40877
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:39725,127.14.67.125:45013,127.14.67.124:45819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:17.279351 22216 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:17.280107 22216 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:17.286008 22224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:17.285975 22221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:17.286222 22222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:17.286477 22216 server_base.cc:1047] running on GCE node
I20250901 20:02:17.287225 22216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:17.287731 22216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:17.288909 22216 hybrid_clock.cc:648] HybridClock initialized: now 1756756937288898 us; error 36 us; skew 500 ppm
Sep 01 20:02:17 dist-test-slave-2s4d krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:17.291504 22216 init.cc:377] Logged in from keytab as oryx/127.14.67.65@KRBTEST.COM (short username oryx)
I20250901 20:02:17.292490 22216 webserver.cc:480] Webserver started at http://127.14.67.65:40877/ using document root <none> and password file <none>
I20250901 20:02:17.292986 22216 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:17.293066 22216 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:17.296301 22216 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20250901 20:02:17.297979 22231 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:17.298992 22216 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250901 20:02:17.299093 22216 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "7d65920bf7264cb5b463e297265d12c4"
format_stamp: "Formatted at 2025-09-01 20:02:14 on dist-test-slave-2s4d"
server_key: "7817388fa48ef330a70bdf8c46220b27"
server_key_iv: "7be44429036a338f3bcc588ce6377aa4"
server_key_version: "encryptionkey@0"
I20250901 20:02:17.299495 22216 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:17.325660 22216 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:17.328336 22216 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:17.328507 22216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:17.329015 22216 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:17.329821 22216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:17.329872 22216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:17.329927 22216 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:17.329959 22216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:17.338894 22216 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:40197
I20250901 20:02:17.338930 22344 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:40197 every 8 connection(s)
I20250901 20:02:17.339819 22216 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:02:17.342761 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 22216
I20250901 20:02:17.342941 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 21557
I20250901 20:02:17.351058 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:35033
--local_ip_for_outbound_sockets=127.14.67.66
--tserver_master_addrs=127.14.67.126:39725
--webserver_port=40739
--webserver_interface=127.14.67.66
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:34599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.67.126:39725,127.14.67.125:45013
--tserver_master_addrs=127.14.67.126:39725,127.14.67.125:45013,127.14.67.124:45819 with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 01 20:02:17 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
Sep 01 20:02:17 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.124@KRBTEST.COM
Sep 01 20:02:17 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.125@KRBTEST.COM
I20250901 20:02:17.359043 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.341361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:38779 (local address 127.14.67.124:45819)
0901 20:02:17.341485 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.341490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.342291 (+ 801us) server_negotiation.cc:407] Connection header received
0901 20:02:17.344979 (+ 2688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.344984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.345071 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.345148 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.347526 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.348353 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.349381 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.349576 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.354400 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.354425 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.354434 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.354478 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.356588 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.357381 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.357387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.357392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.357446 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.357787 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.357790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.357793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.357972 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.358055 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.358746 (+ 691us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.358885 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250901 20:02:17.359045 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.341370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:37037 (local address 127.14.67.126:39725)
0901 20:02:17.341491 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.341495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.342546 (+ 1051us) server_negotiation.cc:407] Connection header received
0901 20:02:17.344982 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.344985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.345094 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.345154 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.347527 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.348367 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.349382 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.349576 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.354048 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.354085 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.354092 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.354129 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.356879 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.357478 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.357484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.357487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.357541 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.357787 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.357790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.357793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.357972 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.358074 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.358746 (+ 672us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.358878 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250901 20:02:17.359848 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.341637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.124:45819 (local address 127.14.67.65:38779)
0901 20:02:17.342158 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.342199 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.344712 (+ 2513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.345430 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.345434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.346356 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:17.347325 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.347347 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.348521 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.348525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.349244 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.349253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.349834 (+ 581us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.351502 (+ 1668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:17.351519 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:17.354259 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:17.356751 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.356759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.356774 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.357238 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.357550 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.357553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.357555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.357670 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.358137 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:17.358144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:17.358424 (+ 280us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.359184 (+ 760us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.359352 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"spinlock_wait_cycles":33152,"thread_start_us":117,"threads_started":1}
I20250901 20:02:17.359848 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.341636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.65:37037)
0901 20:02:17.342445 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.342457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.344712 (+ 2255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.345430 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.345434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.346652 (+ 1218us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:17.347325 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.347347 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.348521 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.348525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.349244 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.349253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.349826 (+ 573us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.351087 (+ 1261us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:17.351102 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:17.353870 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:17.357074 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.357078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.357080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.357368 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.357638 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.357641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.357643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.357687 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.358137 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:17.358144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:17.358424 (+ 280us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.359236 (+ 812us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.359423 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":106,"spinlock_wait_cycles":13056,"thread_start_us":103,"threads_started":1}
I20250901 20:02:17.360013 22351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.341708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.125:45013 (local address 127.14.67.65:60767)
0901 20:02:17.342555 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.342567 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.344646 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.345421 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.345436 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.346420 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:17.347327 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.347348 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.348526 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.348536 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.349315 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.349328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.349867 (+ 539us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.350861 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:17.350898 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:17.354051 (+ 3153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:17.357312 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.357318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.357324 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.357570 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.357864 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.357869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.357874 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.357936 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.358422 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:17.358426 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:17.358563 (+ 137us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.358701 (+ 138us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.359004 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":80,"spinlock_wait_cycles":34816,"thread_start_us":105,"threads_started":1}
I20250901 20:02:17.360770 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.341809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:60767 (local address 127.14.67.125:45013)
0901 20:02:17.341935 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.341939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.342620 (+ 681us) server_negotiation.cc:407] Connection header received
0901 20:02:17.344882 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.344889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.344953 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.345010 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.347508 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.348372 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.349455 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.349670 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.354193 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.354216 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.354222 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.354253 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.357160 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.357697 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.357703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.357706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.357760 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.358042 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.358060 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.358063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.358266 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.358368 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.360455 (+ 2087us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.360573 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250901 20:02:17.360936 22346 heartbeater.cc:344] Connected to a master server at 127.14.67.126:39725
I20250901 20:02:17.361209 22346 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:17.361227 22345 heartbeater.cc:344] Connected to a master server at 127.14.67.124:45819
I20250901 20:02:17.361325 22345 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:17.361627 22347 heartbeater.cc:344] Connected to a master server at 127.14.67.125:45013
I20250901 20:02:17.361707 22347 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:17.361838 22345 heartbeater.cc:507] Master 127.14.67.124:45819 requested a full tablet report, sending...
I20250901 20:02:17.361838 22346 heartbeater.cc:507] Master 127.14.67.126:39725 requested a full tablet report, sending...
I20250901 20:02:17.361932 22347 heartbeater.cc:507] Master 127.14.67.125:45013 requested a full tablet report, sending...
I20250901 20:02:17.362970 21355 ts_manager.cc:194] Re-registered known tserver with Master: 7d65920bf7264cb5b463e297265d12c4 (127.14.67.65:40197)
I20250901 20:02:17.363389 21966 ts_manager.cc:194] Registered new tserver with Master: 7d65920bf7264cb5b463e297265d12c4 (127.14.67.65:40197)
I20250901 20:02:17.363548 21355 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:37037
I20250901 20:02:17.364306 22152 ts_manager.cc:194] Registered new tserver with Master: 7d65920bf7264cb5b463e297265d12c4 (127.14.67.65:40197)
W20250901 20:02:17.458781 22352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:17.458962 22352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:17.459028 22352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:17.462162 22352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:17.462213 22352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:17.462288 22352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:02:17.466320 22352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:34599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:35033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=40739
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:39725,127.14.67.125:45013,127.14.67.124:45819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:17.467288 22352 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:17.468058 22352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:17.473948 22358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:17.473971 22357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:17.474107 22352 server_base.cc:1047] running on GCE node
W20250901 20:02:17.473970 22360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:17.474617 22352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:17.475117 22352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:17.476289 22352 hybrid_clock.cc:648] HybridClock initialized: now 1756756937476273 us; error 34 us; skew 500 ppm
Sep 01 20:02:17 dist-test-slave-2s4d krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:17.478957 22352 init.cc:377] Logged in from keytab as oryx/127.14.67.66@KRBTEST.COM (short username oryx)
I20250901 20:02:17.480000 22352 webserver.cc:480] Webserver started at http://127.14.67.66:40739/ using document root <none> and password file <none>
I20250901 20:02:17.480506 22352 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:17.480564 22352 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:17.484165 22352 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250901 20:02:17.486688 22367 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:17.488019 22352 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250901 20:02:17.488164 22352 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "05d0551bc0f04bdd8fad024b9196fc0d"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "5958e31dc4071729d31534c06ce015e6"
server_key_iv: "1d29797e6c503844cc6d765fb661f183"
server_key_version: "encryptionkey@0"
I20250901 20:02:17.488749 22352 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:17.503903 22352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:17.506636 22352 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:17.506803 22352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:17.507315 22352 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:17.508183 22352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:17.508234 22352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:17.508289 22352 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:17.508323 22352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:17.517189 22352 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:35033
I20250901 20:02:17.517225 22480 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:35033 every 8 connection(s)
I20250901 20:02:17.518172 22352 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
Sep 01 20:02:17 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.124@KRBTEST.COM
Sep 01 20:02:17 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:17.528452 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 22352
Sep 01 20:02:17 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.125@KRBTEST.COM
I20250901 20:02:17.528586 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 21693
I20250901 20:02:17.532733 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.520268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:33263 (local address 127.14.67.126:39725)
0901 20:02:17.520553 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.520561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.521611 (+ 1050us) server_negotiation.cc:407] Connection header received
0901 20:02:17.522147 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.522151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.522220 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.522278 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.523855 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.524398 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.525313 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.525499 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.528138 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.528161 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.528167 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.528201 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.530638 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.531212 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.531216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.531217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.531276 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.531613 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.531619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.531624 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.531850 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.531954 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.532455 (+ 501us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.532573 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20250901 20:02:17.532840 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.520408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:54761 (local address 127.14.67.125:45013)
0901 20:02:17.520658 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.520662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.521565 (+ 903us) server_negotiation.cc:407] Connection header received
0901 20:02:17.522148 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.522151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.522220 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.522278 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.523707 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.524446 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.525334 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.525524 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.529077 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.529101 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.529106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.529140 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.531079 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.531505 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.531509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.531511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.531556 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.531775 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.531779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.531781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.531951 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.532046 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.532478 (+ 432us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.532606 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20250901 20:02:17.533288 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.520301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.66:33263)
0901 20:02:17.521521 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.521535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.521901 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.522443 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.522446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.523458 (+ 1012us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:17.523737 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.523742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.524517 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.524521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.525170 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.525179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.525637 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.526456 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:17.526470 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:17.527975 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:17.530786 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.530793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.530803 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.531100 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.531390 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.531392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.531395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.531505 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.531978 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:17.531987 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:17.532218 (+ 231us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.532402 (+ 184us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.532655 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":372,"spinlock_wait_cycles":1408,"thread_start_us":79,"threads_started":1}
I20250901 20:02:17.533484 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.520301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.125:45013 (local address 127.14.67.66:54761)
0901 20:02:17.521433 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.521450 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.521901 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.522443 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.522446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.522889 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:17.523555 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.523566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.524555 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.524558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.525205 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.525215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.525637 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.526353 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:17.526367 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:17.528883 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:17.531200 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.531203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.531205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.531403 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.531653 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.531655 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.531657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.531692 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.532052 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:17.532058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:17.532229 (+ 171us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.533017 (+ 788us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.533169 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":32,"spinlock_wait_cycles":22144,"thread_start_us":146,"threads_started":1}
I20250901 20:02:17.534348 22483 heartbeater.cc:344] Connected to a master server at 127.14.67.125:45013
I20250901 20:02:17.534436 22482 heartbeater.cc:344] Connected to a master server at 127.14.67.126:39725
I20250901 20:02:17.534600 22482 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:17.534600 22483 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:17.535212 22483 heartbeater.cc:507] Master 127.14.67.125:45013 requested a full tablet report, sending...
I20250901 20:02:17.535212 22482 heartbeater.cc:507] Master 127.14.67.126:39725 requested a full tablet report, sending...
I20250901 20:02:17.536306 21966 ts_manager.cc:194] Registered new tserver with Master: 05d0551bc0f04bdd8fad024b9196fc0d (127.14.67.66:35033)
I20250901 20:02:17.536420 21355 ts_manager.cc:194] Re-registered known tserver with Master: 05d0551bc0f04bdd8fad024b9196fc0d (127.14.67.66:35033)
I20250901 20:02:17.537153 21355 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:33263
I20250901 20:02:17.538615 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:42619
--local_ip_for_outbound_sockets=127.14.67.67
--tserver_master_addrs=127.14.67.126:39725
--webserver_port=38159
--webserver_interface=127.14.67.67
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:34599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.67.126:39725,127.14.67.125:45013
--tserver_master_addrs=127.14.67.126:39725,127.14.67.125:45013,127.14.67.124:45819 with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250901 20:02:17.539635 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.519942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:56319 (local address 127.14.67.124:45819)
0901 20:02:17.520088 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.520092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.521029 (+ 937us) server_negotiation.cc:407] Connection header received
0901 20:02:17.522097 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.522101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.522157 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.522231 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.523616 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.524402 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.525153 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.525326 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.527725 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.527742 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.527745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.527769 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.536466 (+ 8697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.538490 (+ 2024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.538493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.538495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.538575 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.538848 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.538851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.538853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.539064 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.539169 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.539398 (+ 229us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.539503 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20250901 20:02:17.539647 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.520300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.124:45819 (local address 127.14.67.66:56319)
0901 20:02:17.520891 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.520934 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.521851 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.522384 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.522392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.522880 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:17.523473 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.523486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.524543 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.524548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.525049 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.525058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.525247 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.525834 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:17.525858 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:17.527569 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:17.538149 (+ 10580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.538154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.538157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.538377 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.538678 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.538682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.538686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.538750 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.539169 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:17.539173 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:17.539254 (+ 81us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.539343 (+ 89us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.539485 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":138,"threads_started":1}
I20250901 20:02:17.540565 22481 heartbeater.cc:344] Connected to a master server at 127.14.67.124:45819
I20250901 20:02:17.540656 22481 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:17.540812 22481 heartbeater.cc:507] Master 127.14.67.124:45819 requested a full tablet report, sending...
I20250901 20:02:17.541409 22152 ts_manager.cc:194] Registered new tserver with Master: 05d0551bc0f04bdd8fad024b9196fc0d (127.14.67.66:35033)
W20250901 20:02:17.639493 22488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:17.639688 22488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:17.639744 22488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:17.642740 22488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:17.642787 22488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:17.642861 22488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:02:17.646647 22488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:34599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:42619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=38159
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:39725,127.14.67.125:45013,127.14.67.124:45819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:17.647540 22488 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:17.648280 22488 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:17.654340 22496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:17.654376 22494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:17.654376 22493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:17.654479 22488 server_base.cc:1047] running on GCE node
I20250901 20:02:17.655066 22488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:17.655539 22488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:17.656709 22488 hybrid_clock.cc:648] HybridClock initialized: now 1756756937656698 us; error 27 us; skew 500 ppm
Sep 01 20:02:17 dist-test-slave-2s4d krb5kdc[21309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:17.659349 22488 init.cc:377] Logged in from keytab as oryx/127.14.67.67@KRBTEST.COM (short username oryx)
I20250901 20:02:17.660346 22488 webserver.cc:480] Webserver started at http://127.14.67.67:38159/ using document root <none> and password file <none>
I20250901 20:02:17.660847 22488 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:17.660912 22488 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:17.664254 22488 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20250901 20:02:17.666047 22503 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:17.667129 22488 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250901 20:02:17.667222 22488 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "39db649d1bd845a0ac3324d3361819ea"
format_stamp: "Formatted at 2025-09-01 20:02:15 on dist-test-slave-2s4d"
server_key: "c98f8abaa5b3ee9e3ab3d282403f8520"
server_key_iv: "d9ba1f015a70bf4a1599cff704f41886"
server_key_version: "encryptionkey@0"
I20250901 20:02:17.667601 22488 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:17.695318 22488 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:17.697992 22488 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:17.698137 22488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:17.698632 22488 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:17.699416 22488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:17.699462 22488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:17.699501 22488 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:17.699515 22488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:17.708043 22488 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:42619
I20250901 20:02:17.708155 22616 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:42619 every 8 connection(s)
I20250901 20:02:17.709067 22488 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:02:17.714942 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 22488
Sep 01 20:02:17 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.124@KRBTEST.COM
Sep 01 20:02:17 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.125@KRBTEST.COM
Sep 01 20:02:17 dist-test-slave-2s4d krb5kdc[21309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:17.724824 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.711090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.125:45013 (local address 127.14.67.67:48677)
0901 20:02:17.711881 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.711894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.712346 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.712816 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.712823 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.713221 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:17.713744 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.713755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.714776 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.714779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.715330 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.715339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.715765 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.716253 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:17.716269 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:17.718474 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:17.721244 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.721249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.721258 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.721571 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.722733 (+ 1162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.722737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.722739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.722839 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.723249 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:17.723255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:17.723507 (+ 252us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.723713 (+ 206us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.723983 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"spinlock_wait_cycles":111104,"thread_start_us":137,"threads_started":1}
I20250901 20:02:17.725180 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.710818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:57469 (local address 127.14.67.126:39725)
0901 20:02:17.710945 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.710950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.711544 (+ 594us) server_negotiation.cc:407] Connection header received
0901 20:02:17.712543 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.712547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.712597 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.712648 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.714054 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.714580 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.718569 (+ 3989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.718755 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.721233 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.721260 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.721263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.721298 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.723134 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.723629 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.723635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.723638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.723691 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.723987 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.723991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.723993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.724161 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.724269 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.724919 (+ 650us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.725031 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51,"spinlock_wait_cycles":1920}
I20250901 20:02:17.724826 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.710973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.67:57469)
0901 20:02:17.711419 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.711460 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.712333 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.712815 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.712825 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.713705 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:17.713945 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.713952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.714685 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.714688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.717880 (+ 3192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.717892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.718873 (+ 981us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.719622 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:17.719643 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:17.721094 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:17.723274 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.723279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.723282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.723500 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.723799 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.723805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.723810 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.723873 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.724316 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:17.724319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:17.724404 (+ 85us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.724507 (+ 103us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.724624 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":128,"threads_started":1}
I20250901 20:02:17.726402 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.716800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.14.67.126:39725)
0901 20:02:17.717051 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.717055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.717293 (+ 238us) server_negotiation.cc:407] Connection header received
0901 20:02:17.717471 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.717475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.717526 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.717624 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.719311 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.720087 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.721023 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.721239 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.722673 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.722690 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.722693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.722720 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.724516 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.724930 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.724933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.724935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.724972 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.725534 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.725537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.725539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.725677 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.725787 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.726114 (+ 327us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.726217 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"spinlock_wait_cycles":5760,"thread_start_us":63,"threads_started":1}
I20250901 20:02:17.726550 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.710666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:48677 (local address 127.14.67.125:45013)
0901 20:02:17.710795 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.710799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.711960 (+ 1161us) server_negotiation.cc:407] Connection header received
0901 20:02:17.712516 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.712519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.712578 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.712656 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.713885 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.714668 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.715468 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.715649 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.718649 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.718666 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.718670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.718695 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.721103 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.722589 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.722593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.722595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.722637 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.722946 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.722950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.722952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.723115 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.723197 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.726295 (+ 3098us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.726395 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250901 20:02:17.726589 22618 heartbeater.cc:344] Connected to a master server at 127.14.67.126:39725
I20250901 20:02:17.726810 22618 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:17.727152 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.710973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.124:45819 (local address 127.14.67.67:55825)
0901 20:02:17.711863 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.711876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.712346 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.712815 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.712825 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.713244 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:17.713794 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.713804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.714811 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.714814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.715243 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.715251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.715422 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.716034 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:17.716070 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:17.718472 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:17.725162 (+ 6690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.725165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.725167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.725374 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.725761 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:17.725765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:17.725770 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:17.725827 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:17.726689 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:17.726692 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:17.726772 (+ 80us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.726874 (+ 102us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.726985 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":3,"spinlock_wait_cycles":17664,"thread_start_us":138,"threads_started":1}
I20250901 20:02:17.727171 22619 heartbeater.cc:344] Connected to a master server at 127.14.67.125:45013
I20250901 20:02:17.727236 22619 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:17.727357 22618 heartbeater.cc:507] Master 127.14.67.126:39725 requested a full tablet report, sending...
I20250901 20:02:17.727391 22619 heartbeater.cc:507] Master 127.14.67.125:45013 requested a full tablet report, sending...
I20250901 20:02:17.727696 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.710966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:55825 (local address 127.14.67.124:45819)
0901 20:02:17.711109 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.711113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.711958 (+ 845us) server_negotiation.cc:407] Connection header received
0901 20:02:17.712543 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.712547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.712597 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.712648 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.713962 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.714689 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.715352 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.715517 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.718663 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.718677 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.718679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.718705 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.722488 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.725595 (+ 3107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.725598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.725600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.725646 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.725935 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.725938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.725940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.726114 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.726207 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.727434 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.727552 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20250901 20:02:17.728156 22617 heartbeater.cc:344] Connected to a master server at 127.14.67.124:45819
I20250901 20:02:17.728231 22617 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:17.728363 21966 ts_manager.cc:194] Registered new tserver with Master: 39db649d1bd845a0ac3324d3361819ea (127.14.67.67:42619)
I20250901 20:02:17.728438 22617 heartbeater.cc:507] Master 127.14.67.124:45819 requested a full tablet report, sending...
I20250901 20:02:17.729022 22152 ts_manager.cc:194] Registered new tserver with Master: 39db649d1bd845a0ac3324d3361819ea (127.14.67.67:42619)
I20250901 20:02:17.729573 21355 ts_manager.cc:194] Re-registered known tserver with Master: 39db649d1bd845a0ac3324d3361819ea (127.14.67.67:42619)
I20250901 20:02:17.730264 21355 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:57469
I20250901 20:02:17.732823 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.717168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53184 (local address 127.14.67.125:45013)
0901 20:02:17.719416 (+ 2248us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.719422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.719436 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:02:17.719497 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.719501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.719545 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.722159 (+ 2614us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.722322 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.723046 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.725143 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.725429 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.725522 (+ 93us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.725539 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.725541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.725561 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.731091 (+ 5530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.731523 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.731526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.731528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.731571 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.731788 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.731791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.731793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.731951 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.732512 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.732579 (+ 67us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.732679 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2177,"thread_start_us":78,"threads_started":1}
I20250901 20:02:17.731581 21355 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 20:02:17.734053 21355 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:02:17.740370 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.717387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.14.67.124:45819)
0901 20:02:17.717581 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.717585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.719120 (+ 1535us) server_negotiation.cc:407] Connection header received
0901 20:02:17.719327 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.719330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.727836 (+ 8506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.727903 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.728835 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.729544 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.735410 (+ 5866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.735595 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.736747 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.736762 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.736766 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.736790 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.738677 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.739095 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.739098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.739100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.739142 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.739366 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.739369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.739372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.739513 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.739613 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.740105 (+ 492us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.740208 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20250901 20:02:17.752610 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.748691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:35033 (local address 127.0.0.1:41906)
0901 20:02:17.748923 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.748934 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.748992 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.749887 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.749891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.749910 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:17.750198 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.750207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.751626 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.751630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.752213 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.752220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.752328 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.752347 (+ 19us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.752398 (+ 51us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.752461 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":101,"threads_started":1}
I20250901 20:02:17.753697 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.749434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:42619 (local address 127.0.0.1:39096)
0901 20:02:17.749727 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.749744 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.749841 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.750325 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.750328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.750347 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:17.750777 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.750786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.752336 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.752339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.753298 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.753308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.753419 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.753440 (+ 21us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.753485 (+ 45us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.753544 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":118,"threads_started":1}
I20250901 20:02:17.753978 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.748338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:40197 (local address 127.0.0.1:36596)
0901 20:02:17.748615 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.748630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.748715 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.750671 (+ 1956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.750674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.750699 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:17.750942 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.750950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.752688 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.752692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.753536 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.753543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.753613 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.753626 (+ 13us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.753661 (+ 35us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.753713 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
I20250901 20:02:17.754158 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.749154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.14.67.66:35033)
0901 20:02:17.749426 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.749431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.749450 (+ 19us) server_negotiation.cc:407] Connection header received
0901 20:02:17.749531 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.749536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.749725 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.750344 (+ 619us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:17.750586 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.751521 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.753181 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.753602 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.753706 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.753766 (+ 60us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.754033 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
I20250901 20:02:17.754540 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.749595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.14.67.67:42619)
0901 20:02:17.749936 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.749944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.749965 (+ 21us) server_negotiation.cc:407] Connection header received
0901 20:02:17.750046 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.750050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.750188 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.750335 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:17.751018 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.752207 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.753479 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.754132 (+ 653us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.754222 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.754299 (+ 77us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.754394 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":92,"threads_started":1}
I20250901 20:02:17.755268 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.748571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36596 (local address 127.14.67.65:40197)
0901 20:02:17.750096 (+ 1525us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.750102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.750124 (+ 22us) server_negotiation.cc:407] Connection header received
0901 20:02:17.750211 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.750215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.750465 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.750726 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:17.751064 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.752565 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.754271 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.754838 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.754967 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.755034 (+ 67us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.755125 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1427,"thread_start_us":152,"threads_started":1}
I20250901 20:02:17.756757 22551 tablet_service.cc:1468] Processing CreateTablet for tablet bcc5130e45fb46d78a0749cdb7dacd45 (DEFAULT_TABLE table=test-table [id=430a5a9c4f994d449b4a691e2ea8b9dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:17.757516 22551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcc5130e45fb46d78a0749cdb7dacd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:17.757213 22415 tablet_service.cc:1468] Processing CreateTablet for tablet bcc5130e45fb46d78a0749cdb7dacd45 (DEFAULT_TABLE table=test-table [id=430a5a9c4f994d449b4a691e2ea8b9dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:17.757431 22279 tablet_service.cc:1468] Processing CreateTablet for tablet bcc5130e45fb46d78a0749cdb7dacd45 (DEFAULT_TABLE table=test-table [id=430a5a9c4f994d449b4a691e2ea8b9dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:17.758301 22415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcc5130e45fb46d78a0749cdb7dacd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:17.758301 22279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcc5130e45fb46d78a0749cdb7dacd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:17.763384 22648 tablet_bootstrap.cc:492] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea: Bootstrap starting.
I20250901 20:02:17.764971 22650 tablet_bootstrap.cc:492] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4: Bootstrap starting.
I20250901 20:02:17.766551 22648 tablet_bootstrap.cc:654] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:17.766961 22650 tablet_bootstrap.cc:654] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:17.767413 22648 log.cc:826] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:17.768110 22649 tablet_bootstrap.cc:492] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d: Bootstrap starting.
I20250901 20:02:17.768241 22650 log.cc:826] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:17.769181 22648 tablet_bootstrap.cc:492] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea: No bootstrap required, opened a new log
I20250901 20:02:17.769430 22648 ts_tablet_manager.cc:1397] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250901 20:02:17.769852 22650 tablet_bootstrap.cc:492] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4: No bootstrap required, opened a new log
I20250901 20:02:17.770025 22650 ts_tablet_manager.cc:1397] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250901 20:02:17.770373 22649 tablet_bootstrap.cc:654] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:17.771133 22649 log.cc:826] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:17.772214 22648 raft_consensus.cc:357] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:17.772387 22648 raft_consensus.cc:383] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:17.772449 22648 raft_consensus.cc:738] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39db649d1bd845a0ac3324d3361819ea, State: Initialized, Role: FOLLOWER
I20250901 20:02:17.772761 22649 tablet_bootstrap.cc:492] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d: No bootstrap required, opened a new log
I20250901 20:02:17.772846 22648 consensus_queue.cc:260] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:17.772931 22649 ts_tablet_manager.cc:1397] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250901 20:02:17.772822 22650 raft_consensus.cc:357] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:17.773012 22650 raft_consensus.cc:383] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:17.773061 22650 raft_consensus.cc:738] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d65920bf7264cb5b463e297265d12c4, State: Initialized, Role: FOLLOWER
I20250901 20:02:17.773502 22618 heartbeater.cc:499] Master 127.14.67.126:39725 was elected leader, sending a full tablet report...
I20250901 20:02:17.773452 22650 consensus_queue.cc:260] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:17.773662 22648 ts_tablet_manager.cc:1428] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250901 20:02:17.774151 22650 ts_tablet_manager.cc:1428] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250901 20:02:17.774713 22346 heartbeater.cc:499] Master 127.14.67.126:39725 was elected leader, sending a full tablet report...
I20250901 20:02:17.780397 22649 raft_consensus.cc:357] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:17.780567 22649 raft_consensus.cc:383] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:17.780599 22649 raft_consensus.cc:738] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d0551bc0f04bdd8fad024b9196fc0d, State: Initialized, Role: FOLLOWER
I20250901 20:02:17.780934 22649 consensus_queue.cc:260] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:17.781620 22649 ts_tablet_manager.cc:1428] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250901 20:02:17.781684 22482 heartbeater.cc:499] Master 127.14.67.126:39725 was elected leader, sending a full tablet report...
W20250901 20:02:17.841194 22348 tablet.cc:2378] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 20:02:17.867141 22654 raft_consensus.cc:491] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:02:17.867360 22654 raft_consensus.cc:513] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:17.868506 22654 leader_election.cc:290] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d65920bf7264cb5b463e297265d12c4 (127.14.67.65:40197), 05d0551bc0f04bdd8fad024b9196fc0d (127.14.67.66:35033)
I20250901 20:02:17.871635 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.868692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:40197 (local address 127.14.67.67:40965)
0901 20:02:17.868910 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.868926 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.869028 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.869364 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.869366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.869382 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:17.869569 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.869574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.870639 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.870643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.871245 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.871253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.871374 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.871396 (+ 22us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.871448 (+ 52us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.871512 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20250901 20:02:17.872061 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.868857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:35033 (local address 127.14.67.67:38629)
0901 20:02:17.869196 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:17.869210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:17.869268 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:17.869643 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:17.869649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:17.869672 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:17.869874 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.869880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.870905 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.870909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:17.871572 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:17.871581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.871702 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.871727 (+ 25us) client_negotiation.cc:769] Sending connection context
0901 20:02:17.871780 (+ 53us) client_negotiation.cc:240] Negotiation successful
0901 20:02:17.871894 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20250901 20:02:17.872220 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.868902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:40965 (local address 127.14.67.65:40197)
0901 20:02:17.869051 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.869055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.869072 (+ 17us) server_negotiation.cc:407] Connection header received
0901 20:02:17.869129 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.869134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.869183 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.869259 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:17.869729 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.870378 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.871369 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.871910 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.871949 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.871997 (+ 48us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.872079 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20250901 20:02:17.872529 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.869235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:38629 (local address 127.14.67.66:35033)
0901 20:02:17.869406 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.869410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.869424 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:02:17.869480 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.869482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.869528 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.869605 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:17.870014 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.870734 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.871695 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.872242 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.872289 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.872344 (+ 55us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.872414 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250901 20:02:17.872831 22299 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcc5130e45fb46d78a0749cdb7dacd45" candidate_uuid: "39db649d1bd845a0ac3324d3361819ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d65920bf7264cb5b463e297265d12c4" is_pre_election: true
I20250901 20:02:17.873127 22299 raft_consensus.cc:2466] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39db649d1bd845a0ac3324d3361819ea in term 0.
I20250901 20:02:17.873198 22435 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcc5130e45fb46d78a0749cdb7dacd45" candidate_uuid: "39db649d1bd845a0ac3324d3361819ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" is_pre_election: true
I20250901 20:02:17.873457 22435 raft_consensus.cc:2466] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39db649d1bd845a0ac3324d3361819ea in term 0.
I20250901 20:02:17.873591 22505 leader_election.cc:304] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39db649d1bd845a0ac3324d3361819ea, 7d65920bf7264cb5b463e297265d12c4; no voters:
I20250901 20:02:17.873852 22654 raft_consensus.cc:2802] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:02:17.873912 22654 raft_consensus.cc:491] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:02:17.873970 22654 raft_consensus.cc:3058] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:17.874946 22654 raft_consensus.cc:513] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:17.875272 22654 leader_election.cc:290] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [CANDIDATE]: Term 1 election: Requested vote from peers 7d65920bf7264cb5b463e297265d12c4 (127.14.67.65:40197), 05d0551bc0f04bdd8fad024b9196fc0d (127.14.67.66:35033)
I20250901 20:02:17.875710 22299 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcc5130e45fb46d78a0749cdb7dacd45" candidate_uuid: "39db649d1bd845a0ac3324d3361819ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d65920bf7264cb5b463e297265d12c4"
I20250901 20:02:17.875710 22435 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcc5130e45fb46d78a0749cdb7dacd45" candidate_uuid: "39db649d1bd845a0ac3324d3361819ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d0551bc0f04bdd8fad024b9196fc0d"
I20250901 20:02:17.875829 22299 raft_consensus.cc:3058] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:17.875842 22435 raft_consensus.cc:3058] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:17.876947 22299 raft_consensus.cc:2466] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39db649d1bd845a0ac3324d3361819ea in term 1.
I20250901 20:02:17.876947 22435 raft_consensus.cc:2466] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39db649d1bd845a0ac3324d3361819ea in term 1.
I20250901 20:02:17.877270 22507 leader_election.cc:304] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d0551bc0f04bdd8fad024b9196fc0d, 39db649d1bd845a0ac3324d3361819ea; no voters:
I20250901 20:02:17.877437 22654 raft_consensus.cc:2802] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:17.877641 22654 raft_consensus.cc:695] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 1 LEADER]: Becoming Leader. State: Replica: 39db649d1bd845a0ac3324d3361819ea, State: Running, Role: LEADER
I20250901 20:02:17.877923 22654 consensus_queue.cc:237] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:17.881474 21355 catalog_manager.cc:5582] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 39db649d1bd845a0ac3324d3361819ea (127.14.67.67). New cstate: current_term: 1 leader_uuid: "39db649d1bd845a0ac3324d3361819ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } health_report { overall_health: HEALTHY } } }
I20250901 20:02:17.907661 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.904337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.14.67.67:42619)
0901 20:02:17.904514 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.904519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.904538 (+ 19us) server_negotiation.cc:407] Connection header received
0901 20:02:17.904648 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.904651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.904716 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.904810 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:02:17.905218 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.905748 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.906377 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.906611 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.906716 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:02:17.907253 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:02:17.907366 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.907445 (+ 79us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.907497 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20250901 20:02:17.913385 22299 raft_consensus.cc:1273] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4 [term 1 FOLLOWER]: Refusing update from remote peer 39db649d1bd845a0ac3324d3361819ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:17.913385 22435 raft_consensus.cc:1273] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d [term 1 FOLLOWER]: Refusing update from remote peer 39db649d1bd845a0ac3324d3361819ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:17.914441 22657 consensus_queue.cc:1035] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250901 20:02:17.914646 22654 consensus_queue.cc:1035] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:17.924840 22662 mvcc.cc:204] Tried to move back new op lower bound from 7195676417686839296 to 7195676417550528512. Current Snapshot: MvccSnapshot[applied={T|T < 7195676417686839296}]
I20250901 20:02:17.925957 22661 mvcc.cc:204] Tried to move back new op lower bound from 7195676417686839296 to 7195676417550528512. Current Snapshot: MvccSnapshot[applied={T|T < 7195676417686839296}]
I20250901 20:02:17.933902 21354 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35964:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:17.934098 21354 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35964:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:17.938217 21354 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 92c3ebb531304adab92f7135e8afbb43: Sending DeleteTablet for 3 replicas of tablet bcc5130e45fb46d78a0749cdb7dacd45
I20250901 20:02:17.938958 22279 tablet_service.cc:1515] Processing DeleteTablet for tablet bcc5130e45fb46d78a0749cdb7dacd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:17 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:36596
I20250901 20:02:17.939559 22670 tablet_replica.cc:331] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4: stopping tablet replica
I20250901 20:02:17.939534 22415 tablet_service.cc:1515] Processing DeleteTablet for tablet bcc5130e45fb46d78a0749cdb7dacd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:17 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:41906
I20250901 20:02:17.939728 22551 tablet_service.cc:1515] Processing DeleteTablet for tablet bcc5130e45fb46d78a0749cdb7dacd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:17 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:39096
I20250901 20:02:17.939975 22670 raft_consensus.cc:2241] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:17.940146 22672 tablet_replica.cc:331] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea: stopping tablet replica
I20250901 20:02:17.940343 22672 raft_consensus.cc:2241] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:02:17.940394 22670 raft_consensus.cc:2270] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:17.940698 22672 raft_consensus.cc:2270] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:17.941895 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 21325
I20250901 20:02:17.942658 22672 ts_tablet_manager.cc:1905] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:17.944526 22672 ts_tablet_manager.cc:1918] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:17.944607 22672 log.cc:1199] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/bcc5130e45fb46d78a0749cdb7dacd45
I20250901 20:02:17.944875 22672 ts_tablet_manager.cc:1939] T bcc5130e45fb46d78a0749cdb7dacd45 P 39db649d1bd845a0ac3324d3361819ea: Deleting consensus metadata
I20250901 20:02:17.947156 22670 ts_tablet_manager.cc:1905] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:17.947474 22671 tablet_replica.cc:331] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d: stopping tablet replica
I20250901 20:02:17.947760 22671 raft_consensus.cc:2241] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:17.948092 22671 raft_consensus.cc:2270] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:17.949484 22671 ts_tablet_manager.cc:1905] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:17.949635 22670 ts_tablet_manager.cc:1918] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:17.949728 22670 log.cc:1199] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-0/wal/wals/bcc5130e45fb46d78a0749cdb7dacd45
I20250901 20:02:17.950063 22670 ts_tablet_manager.cc:1939] T bcc5130e45fb46d78a0749cdb7dacd45 P 7d65920bf7264cb5b463e297265d12c4: Deleting consensus metadata
W20250901 20:02:17.949136 22505 connection.cc:537] server connection from 127.0.0.1:39096 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 20:02:17.951510 22234 connection.cc:441] server connection from 127.0.0.1:36596 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36596 (request call id 1) could send its response
I20250901 20:02:17.952283 22671 ts_tablet_manager.cc:1918] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:17.952368 22671 log.cc:1199] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/bcc5130e45fb46d78a0749cdb7dacd45
I20250901 20:02:17.952693 22671 ts_tablet_manager.cc:1939] T bcc5130e45fb46d78a0749cdb7dacd45 P 05d0551bc0f04bdd8fad024b9196fc0d: Deleting consensus metadata
W20250901 20:02:17.953586 22371 connection.cc:441] server connection from 127.0.0.1:41906 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41906 (request call id 1) could send its response
I20250901 20:02:17.961673 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.953901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.14.67.124:45819)
0901 20:02:17.954027 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.954032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.954330 (+ 298us) server_negotiation.cc:407] Connection header received
0901 20:02:17.954549 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.954553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.954610 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.954667 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.955664 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.956156 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.956906 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.957117 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.958345 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.958361 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.958364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.958389 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.960254 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.960596 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.960600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.960602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.960644 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.960840 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.960844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.960845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.960975 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.961098 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.961296 (+ 198us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.961509 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250901 20:02:17.962975 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:17.954333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.14.67.125:45013)
0901 20:02:17.954445 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:17.954450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:17.954813 (+ 363us) server_negotiation.cc:407] Connection header received
0901 20:02:17.954917 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:17.954920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:17.954975 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:17.955073 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:17.956472 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.957159 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:17.958003 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:17.958214 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:17.959410 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:17.959427 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:17.959430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:17.959455 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:17.961502 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.961929 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.961934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.961938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.961984 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:17.962234 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:17.962238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:17.962240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:17.962404 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:17.962557 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:17.962719 (+ 162us) server_negotiation.cc:299] Negotiation successful
0901 20:02:17.962824 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20250901 20:02:18.927871 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:18.927351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.65:46755)
0901 20:02:18.927670 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:18.927744 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":117,"threads_started":1}
W20250901 20:02:18.928275 22346 heartbeater.cc:646] Failed to heartbeat to 127.14.67.126:39725 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
I20250901 20:02:18.928544 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:18.928021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.67:40467)
0901 20:02:18.928313 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:18.928409 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
W20250901 20:02:18.929186 22618 heartbeater.cc:646] Failed to heartbeat to 127.14.67.126:39725 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
I20250901 20:02:18.929668 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:18.929232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.66:47105)
0901 20:02:18.929499 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:18.929572 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
W20250901 20:02:18.930370 22482 heartbeater.cc:646] Failed to heartbeat to 127.14.67.126:39725 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
I20250901 20:02:19.438702 22683 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92c3ebb531304adab92f7135e8afbb43)
I20250901 20:02:19.438901 22683 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } }
I20250901 20:02:19.439183 22684 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92c3ebb531304adab92f7135e8afbb43)
I20250901 20:02:19.439329 22684 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } }
I20250901 20:02:19.439921 22683 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725), 1c0d1c7626a0438b866868946ac054e2 (127.14.67.124:45819)
I20250901 20:02:19.440126 22684 leader_election.cc:290] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725), 5f0800e54af54372b1e66ac2c3f428ea (127.14.67.125:45013)
I20250901 20:02:19.440467 22162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0800e54af54372b1e66ac2c3f428ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1c0d1c7626a0438b866868946ac054e2" is_pre_election: true
I20250901 20:02:19.440469 21978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c0d1c7626a0438b866868946ac054e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f0800e54af54372b1e66ac2c3f428ea" is_pre_election: true
I20250901 20:02:19.440518 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:19.440071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36436)
0901 20:02:19.440352 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:19.440419 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":61,"threads_started":1}
I20250901 20:02:19.440634 21978 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0d1c7626a0438b866868946ac054e2 in term 1.
I20250901 20:02:19.440615 22162 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f0800e54af54372b1e66ac2c3f428ea in term 1.
I20250901 20:02:19.440625 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:19.440195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36442)
0901 20:02:19.440464 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:19.440531 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":55,"threads_started":1}
W20250901 20:02:19.440774 21955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
W20250901 20:02:19.440855 22139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
I20250901 20:02:19.440989 22138 leader_election.cc:304] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0d1c7626a0438b866868946ac054e2, 5f0800e54af54372b1e66ac2c3f428ea; no voters:
I20250901 20:02:19.441190 22684 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250901 20:02:19.441253 22684 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92c3ebb531304adab92f7135e8afbb43)
I20250901 20:02:19.441298 22684 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 1 FOLLOWER]: Advancing to term 2
I20250901 20:02:19.441279 21953 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0d1c7626a0438b866868946ac054e2, 5f0800e54af54372b1e66ac2c3f428ea; no voters:
I20250901 20:02:19.441450 22683 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250901 20:02:19.441753 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:19.441520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36454)
0901 20:02:19.441658 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:19.441698 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20250901 20:02:19.441764 22683 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92c3ebb531304adab92f7135e8afbb43)
W20250901 20:02:19.441856 21955 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725): Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
I20250901 20:02:19.441846 22683 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 1 FOLLOWER]: Advancing to term 2
I20250901 20:02:19.442400 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:19.442117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36456)
0901 20:02:19.442289 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:19.442335 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20250901 20:02:19.442421 22684 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } }
W20250901 20:02:19.442484 22139 leader_election.cc:336] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725): Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
I20250901 20:02:19.442798 22684 leader_election.cc:290] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [CANDIDATE]: Term 2 election: Requested vote from peers 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725), 5f0800e54af54372b1e66ac2c3f428ea (127.14.67.125:45013)
I20250901 20:02:19.443176 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:19.442898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36470)
0901 20:02:19.443085 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:19.443131 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20250901 20:02:19.443145 21978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c0d1c7626a0438b866868946ac054e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f0800e54af54372b1e66ac2c3f428ea"
I20250901 20:02:19.443310 22683 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } }
I20250901 20:02:19.443542 21978 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c0d1c7626a0438b866868946ac054e2 in current term 2: Already voted for candidate 5f0800e54af54372b1e66ac2c3f428ea in this term.
I20250901 20:02:19.443892 22683 leader_election.cc:290] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [CANDIDATE]: Term 2 election: Requested vote from peers 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725), 1c0d1c7626a0438b866868946ac054e2 (127.14.67.124:45819)
I20250901 20:02:19.444217 22162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0800e54af54372b1e66ac2c3f428ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1c0d1c7626a0438b866868946ac054e2"
I20250901 20:02:19.444350 22162 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f0800e54af54372b1e66ac2c3f428ea in current term 2: Already voted for candidate 1c0d1c7626a0438b866868946ac054e2 in this term.
I20250901 20:02:19.444540 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:19.443734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36474)
0901 20:02:19.444129 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:19.444471 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20250901 20:02:19.444617 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:19.444352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36488)
0901 20:02:19.444503 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:19.444553 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20250901 20:02:19.444716 22139 leader_election.cc:336] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725): Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
I20250901 20:02:19.444792 22139 leader_election.cc:304] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c0d1c7626a0438b866868946ac054e2; no voters: 5f0800e54af54372b1e66ac2c3f428ea, 92c3ebb531304adab92f7135e8afbb43
I20250901 20:02:19.444911 22684 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250901 20:02:19.445169 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:19.444920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36504)
0901 20:02:19.445080 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:19.445118 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
W20250901 20:02:19.445272 21955 leader_election.cc:336] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725): Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
I20250901 20:02:19.445343 21955 leader_election.cc:304] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0800e54af54372b1e66ac2c3f428ea; no voters: 1c0d1c7626a0438b866868946ac054e2, 92c3ebb531304adab92f7135e8afbb43
I20250901 20:02:19.445459 22683 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250901 20:02:19.929517 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:19.928891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.65:38557)
0901 20:02:19.929289 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:19.929372 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20250901 20:02:19.930310 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:19.929746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.67:39143)
0901 20:02:19.930145 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:19.930208 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":95,"threads_started":1}
I20250901 20:02:19.931959 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:19.930858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.66:59579)
0901 20:02:19.931777 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:19.931852 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":77,"threads_started":1}
I20250901 20:02:20.931478 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:20.930682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.65:40581)
0901 20:02:20.931272 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:20.931346 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":136,"threads_started":1}
I20250901 20:02:20.933765 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:20.933282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.67:56103)
0901 20:02:20.933631 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:20.933682 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":80,"threads_started":1}
I20250901 20:02:20.933892 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:20.932628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.66:38735)
0901 20:02:20.933655 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:20.933732 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":80,"threads_started":1}
I20250901 20:02:21.089269 22696 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:02:21.089454 22696 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } }
I20250901 20:02:21.089977 22696 leader_election.cc:290] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725), 5f0800e54af54372b1e66ac2c3f428ea (127.14.67.125:45013)
I20250901 20:02:21.090605 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.090184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36728)
0901 20:02:21.090396 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.090456 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":56,"threads_started":1}
I20250901 20:02:21.090543 21978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c0d1c7626a0438b866868946ac054e2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f0800e54af54372b1e66ac2c3f428ea" is_pre_election: true
I20250901 20:02:21.090715 21978 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c0d1c7626a0438b866868946ac054e2 in term 2.
I20250901 20:02:21.091087 22138 leader_election.cc:304] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0d1c7626a0438b866868946ac054e2, 5f0800e54af54372b1e66ac2c3f428ea; no voters:
I20250901 20:02:21.091276 22696 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250901 20:02:21.091331 22696 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:02:21.091367 22696 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 2 FOLLOWER]: Advancing to term 3
I20250901 20:02:21.091558 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.091317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36738)
0901 20:02:21.091447 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.091497 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20250901 20:02:21.091677 22139 leader_election.cc:336] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725): Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
I20250901 20:02:21.092442 22696 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } }
I20250901 20:02:21.092765 22696 leader_election.cc:290] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [CANDIDATE]: Term 3 election: Requested vote from peers 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725), 5f0800e54af54372b1e66ac2c3f428ea (127.14.67.125:45013)
I20250901 20:02:21.093114 21978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c0d1c7626a0438b866868946ac054e2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f0800e54af54372b1e66ac2c3f428ea"
I20250901 20:02:21.093204 21978 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 2 FOLLOWER]: Advancing to term 3
I20250901 20:02:21.093211 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.092949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36740)
0901 20:02:21.093098 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.093141 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20250901 20:02:21.093880 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.093713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36746)
0901 20:02:21.093800 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.093838 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20250901 20:02:21.094005 22139 leader_election.cc:336] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725): Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
I20250901 20:02:21.094292 21978 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c0d1c7626a0438b866868946ac054e2 in term 3.
I20250901 20:02:21.094609 22138 leader_election.cc:304] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c0d1c7626a0438b866868946ac054e2, 5f0800e54af54372b1e66ac2c3f428ea; no voters: 92c3ebb531304adab92f7135e8afbb43
I20250901 20:02:21.094774 22696 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 3 FOLLOWER]: Leader election won for term 3
I20250901 20:02:21.095026 22696 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [term 3 LEADER]: Becoming Leader. State: Replica: 1c0d1c7626a0438b866868946ac054e2, State: Running, Role: LEADER
I20250901 20:02:21.095300 22696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } }
I20250901 20:02:21.096549 22699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c0d1c7626a0438b866868946ac054e2. Latest consensus state: current_term: 3 leader_uuid: "1c0d1c7626a0438b866868946ac054e2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } } }
I20250901 20:02:21.096647 22699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:21.096947 22701 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:02:21.099188 22701 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:02:21.099565 22701 catalog_manager.cc:1261] Loaded cluster ID: edd92a8d72a04b938d695a74fda08800
I20250901 20:02:21.099610 22701 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:02:21.100095 22701 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:02:21.100440 22701 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: Loaded TSK: 0
I20250901 20:02:21.101053 22701 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 20:02:21.159170 22151 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 20:02:21.161651 22151 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:02:21.164312 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.164094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36764)
0901 20:02:21.164208 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.164250 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20250901 20:02:21.164356 21978 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [term 3 FOLLOWER]: Refusing update from remote peer 1c0d1c7626a0438b866868946ac054e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250901 20:02:21.164794 22699 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250901 20:02:21.164934 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.164751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36770)
0901 20:02:21.164850 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.164890 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20250901 20:02:21.165036 22139 consensus_peers.cc:489] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 -> Peer 92c3ebb531304adab92f7135e8afbb43 (127.14.67.126:39725): Couldn't send request to peer 92c3ebb531304adab92f7135e8afbb43. Status: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 20:02:21.166080 22703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c0d1c7626a0438b866868946ac054e2. Latest consensus state: current_term: 3 leader_uuid: "1c0d1c7626a0438b866868946ac054e2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } } }
I20250901 20:02:21.166181 22703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: This master's current role is: FOLLOWER
I20250901 20:02:21.167148 22699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "1c0d1c7626a0438b866868946ac054e2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } } }
I20250901 20:02:21.167251 22699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:21.169045 22703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1c0d1c7626a0438b866868946ac054e2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92c3ebb531304adab92f7135e8afbb43" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39725 } } peers { permanent_uuid: "5f0800e54af54372b1e66ac2c3f428ea" member_type: VOTER last_known_addr { host: "127.14.67.125" port: 45013 } attrs { promote: false } } peers { permanent_uuid: "1c0d1c7626a0438b866868946ac054e2" member_type: VOTER last_known_addr { host: "127.14.67.124" port: 45819 } attrs { promote: false } } }
I20250901 20:02:21.169207 22703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0800e54af54372b1e66ac2c3f428ea [sys.catalog]: This master's current role is: FOLLOWER
I20250901 20:02:21.176134 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.172332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:40197 (local address 127.0.0.1:36610)
0901 20:02:21.172543 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.172558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:21.172655 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:21.173345 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:21.173347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:21.173357 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:21.173618 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:21.173625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.174781 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:21.174786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:21.175753 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:21.175763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.175872 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:21.175891 (+ 19us) client_negotiation.cc:769] Sending connection context
0901 20:02:21.175940 (+ 49us) client_negotiation.cc:240] Negotiation successful
0901 20:02:21.176008 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20250901 20:02:21.176265 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.172061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:35033 (local address 127.0.0.1:41920)
0901 20:02:21.172183 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.172198 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:21.172283 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:21.173777 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:21.173782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:21.173802 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:21.174055 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:21.174063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.175041 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:21.175044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:21.175945 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:21.175953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.176057 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:21.176070 (+ 13us) client_negotiation.cc:769] Sending connection context
0901 20:02:21.176113 (+ 43us) client_negotiation.cc:240] Negotiation successful
0901 20:02:21.176173 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20250901 20:02:21.177137 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.172880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.14.67.65:40197)
0901 20:02:21.173132 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:21.173135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:21.173146 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:02:21.173194 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:21.173197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:21.173250 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:21.173345 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:21.173832 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.174637 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:21.176307 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.176877 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:21.176908 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:21.176956 (+ 48us) server_negotiation.cc:299] Negotiation successful
0901 20:02:21.177007 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":100,"threads_started":1}
I20250901 20:02:21.177168 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.172243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.14.67.66:35033)
0901 20:02:21.173553 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:21.173557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:21.173571 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:02:21.173612 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:21.173614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:21.173652 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:21.174158 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:21.174251 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.174934 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:21.176403 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.176872 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:21.176908 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:21.176956 (+ 48us) server_negotiation.cc:299] Negotiation successful
0901 20:02:21.177022 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1233,"thread_start_us":77,"threads_started":1}
I20250901 20:02:21.177606 22415 tablet_service.cc:1468] Processing CreateTablet for tablet 5675f84b310c4ed499457eab360b9e3f (DEFAULT_TABLE table=test-table [id=66b5b8253d97435c88b8e419faf7fcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:21.177640 22279 tablet_service.cc:1468] Processing CreateTablet for tablet 5675f84b310c4ed499457eab360b9e3f (DEFAULT_TABLE table=test-table [id=66b5b8253d97435c88b8e419faf7fcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:21.177878 22279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5675f84b310c4ed499457eab360b9e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:21.177881 22415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5675f84b310c4ed499457eab360b9e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:21.178611 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.172660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:42619 (local address 127.0.0.1:39110)
0901 20:02:21.173119 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.173134 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:21.173209 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:21.173448 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:21.173452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:21.173465 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:21.173661 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:21.173670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.177486 (+ 3816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:21.177490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:21.178238 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:21.178262 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.178375 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:21.178390 (+ 15us) client_negotiation.cc:769] Sending connection context
0901 20:02:21.178435 (+ 45us) client_negotiation.cc:240] Negotiation successful
0901 20:02:21.178483 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":77,"threads_started":1}
I20250901 20:02:21.179229 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.172787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.14.67.67:42619)
0901 20:02:21.173043 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:21.173048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:21.173194 (+ 146us) server_negotiation.cc:407] Connection header received
0901 20:02:21.173283 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:21.173289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:21.173350 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:21.173433 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:21.173788 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.174594 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:21.178388 (+ 3794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.178908 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:21.178950 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:21.179016 (+ 66us) server_negotiation.cc:299] Negotiation successful
0901 20:02:21.179086 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20250901 20:02:21.180040 22714 tablet_bootstrap.cc:492] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d: Bootstrap starting.
I20250901 20:02:21.180450 22715 tablet_bootstrap.cc:492] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4: Bootstrap starting.
I20250901 20:02:21.180464 22551 tablet_service.cc:1468] Processing CreateTablet for tablet 5675f84b310c4ed499457eab360b9e3f (DEFAULT_TABLE table=test-table [id=66b5b8253d97435c88b8e419faf7fcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:21.180713 22551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5675f84b310c4ed499457eab360b9e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:21.181241 22714 tablet_bootstrap.cc:654] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:21.181602 22715 tablet_bootstrap.cc:654] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:21.182325 22714 tablet_bootstrap.cc:492] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d: No bootstrap required, opened a new log
I20250901 20:02:21.182381 22714 ts_tablet_manager.cc:1397] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:21.182647 22718 tablet_bootstrap.cc:492] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea: Bootstrap starting.
I20250901 20:02:21.183100 22714 raft_consensus.cc:357] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:21.183209 22714 raft_consensus.cc:383] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:21.183252 22714 raft_consensus.cc:738] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d0551bc0f04bdd8fad024b9196fc0d, State: Initialized, Role: FOLLOWER
I20250901 20:02:21.183379 22714 consensus_queue.cc:260] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:21.183522 22718 tablet_bootstrap.cc:654] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:21.183779 22714 ts_tablet_manager.cc:1428] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:02:21.184474 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.184197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.66:36739)
0901 20:02:21.184358 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.184407 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20250901 20:02:21.185561 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.185365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.66:46915)
0901 20:02:21.185466 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.185515 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20250901 20:02:21.185884 22481 heartbeater.cc:499] Master 127.14.67.124:45819 was elected leader, sending a full tablet report...
I20250901 20:02:21.186174 22715 tablet_bootstrap.cc:492] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4: No bootstrap required, opened a new log
I20250901 20:02:21.186280 22715 ts_tablet_manager.cc:1397] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20250901 20:02:21.186777 22715 raft_consensus.cc:357] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:21.186908 22715 raft_consensus.cc:383] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:21.186947 22715 raft_consensus.cc:738] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d65920bf7264cb5b463e297265d12c4, State: Initialized, Role: FOLLOWER
I20250901 20:02:21.187091 22715 consensus_queue.cc:260] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:21.187475 22715 ts_tablet_manager.cc:1428] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250901 20:02:21.188064 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.187816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.65:37725)
0901 20:02:21.187932 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.187984 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20250901 20:02:21.188388 22718 tablet_bootstrap.cc:492] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea: No bootstrap required, opened a new log
I20250901 20:02:21.188607 22718 ts_tablet_manager.cc:1397] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20250901 20:02:21.188879 22345 heartbeater.cc:499] Master 127.14.67.124:45819 was elected leader, sending a full tablet report...
I20250901 20:02:21.189065 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.188834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.65:51861)
0901 20:02:21.188942 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.188997 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20250901 20:02:21.189039 22718 raft_consensus.cc:357] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:21.189136 22718 raft_consensus.cc:383] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:21.189172 22718 raft_consensus.cc:738] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39db649d1bd845a0ac3324d3361819ea, State: Initialized, Role: FOLLOWER
I20250901 20:02:21.189945 22718 consensus_queue.cc:260] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:21.190610 22718 ts_tablet_manager.cc:1428] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250901 20:02:21.190758 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.190560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.67:49835)
0901 20:02:21.190662 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.190705 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20250901 20:02:21.191350 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.191144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.67:51217)
0901 20:02:21.191273 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.191308 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20250901 20:02:21.191919 22617 heartbeater.cc:499] Master 127.14.67.124:45819 was elected leader, sending a full tablet report...
W20250901 20:02:21.213203 22620 tablet.cc:2378] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250901 20:02:21.273703 22484 tablet.cc:2378] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 20:02:21.590026 22719 raft_consensus.cc:491] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:02:21.590181 22719 raft_consensus.cc:513] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:21.591336 22719 leader_election.cc:290] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d65920bf7264cb5b463e297265d12c4 (127.14.67.65:40197), 39db649d1bd845a0ac3324d3361819ea (127.14.67.67:42619)
I20250901 20:02:21.594388 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.591595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:40197 (local address 127.14.67.66:43677)
0901 20:02:21.591708 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.591723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:21.591852 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:21.592102 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:21.592105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:21.592121 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:21.592341 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:21.592347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.593221 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:21.593225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:21.594013 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:21.594022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.594135 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:21.594156 (+ 21us) client_negotiation.cc:769] Sending connection context
0901 20:02:21.594204 (+ 48us) client_negotiation.cc:240] Negotiation successful
0901 20:02:21.594267 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20250901 20:02:21.594769 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.591595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:42619 (local address 127.14.67.66:47387)
0901 20:02:21.592262 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.592274 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:21.592360 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:21.592611 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:21.592614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:21.592631 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:21.592865 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:21.592872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.593574 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:21.593578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:21.594380 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:21.594388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.594493 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:21.594536 (+ 43us) client_negotiation.cc:769] Sending connection context
0901 20:02:21.594591 (+ 55us) client_negotiation.cc:240] Negotiation successful
0901 20:02:21.594641 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":106,"thread_start_us":149,"threads_started":1}
I20250901 20:02:21.594871 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.591681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:43677 (local address 127.14.67.65:40197)
0901 20:02:21.591843 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:21.591847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:21.591864 (+ 17us) server_negotiation.cc:407] Connection header received
0901 20:02:21.591960 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:21.591964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:21.592007 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:21.592083 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:21.592448 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.593119 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:21.594161 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.594585 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:21.594629 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:21.594681 (+ 52us) server_negotiation.cc:299] Negotiation successful
0901 20:02:21.594746 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20250901 20:02:21.595242 22299 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5675f84b310c4ed499457eab360b9e3f" candidate_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d65920bf7264cb5b463e297265d12c4" is_pre_election: true
I20250901 20:02:21.595356 22299 raft_consensus.cc:2466] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05d0551bc0f04bdd8fad024b9196fc0d in term 0.
I20250901 20:02:21.595525 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.591757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:47387 (local address 127.14.67.67:42619)
0901 20:02:21.591892 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:21.591899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:21.592344 (+ 445us) server_negotiation.cc:407] Connection header received
0901 20:02:21.592486 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:21.592489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:21.592529 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:21.592588 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:21.592976 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.593480 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:21.594763 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.595249 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:21.595279 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:21.595323 (+ 44us) server_negotiation.cc:299] Negotiation successful
0901 20:02:21.595389 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20250901 20:02:21.595800 22369 leader_election.cc:304] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d0551bc0f04bdd8fad024b9196fc0d, 7d65920bf7264cb5b463e297265d12c4; no voters:
I20250901 20:02:21.595965 22571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5675f84b310c4ed499457eab360b9e3f" candidate_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39db649d1bd845a0ac3324d3361819ea" is_pre_election: true
I20250901 20:02:21.596020 22719 raft_consensus.cc:2802] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:02:21.596069 22719 raft_consensus.cc:491] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:02:21.596107 22719 raft_consensus.cc:3058] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:21.596163 22571 raft_consensus.cc:2466] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05d0551bc0f04bdd8fad024b9196fc0d in term 0.
I20250901 20:02:21.597088 22719 raft_consensus.cc:513] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:21.597442 22719 leader_election.cc:290] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [CANDIDATE]: Term 1 election: Requested vote from peers 7d65920bf7264cb5b463e297265d12c4 (127.14.67.65:40197), 39db649d1bd845a0ac3324d3361819ea (127.14.67.67:42619)
I20250901 20:02:21.597762 22299 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5675f84b310c4ed499457eab360b9e3f" candidate_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d65920bf7264cb5b463e297265d12c4"
I20250901 20:02:21.597762 22571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5675f84b310c4ed499457eab360b9e3f" candidate_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39db649d1bd845a0ac3324d3361819ea"
I20250901 20:02:21.597872 22571 raft_consensus.cc:3058] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:21.597872 22299 raft_consensus.cc:3058] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:21.598891 22571 raft_consensus.cc:2466] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05d0551bc0f04bdd8fad024b9196fc0d in term 1.
I20250901 20:02:21.598891 22299 raft_consensus.cc:2466] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05d0551bc0f04bdd8fad024b9196fc0d in term 1.
I20250901 20:02:21.599180 22370 leader_election.cc:304] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d0551bc0f04bdd8fad024b9196fc0d, 39db649d1bd845a0ac3324d3361819ea; no voters:
I20250901 20:02:21.599336 22719 raft_consensus.cc:2802] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:21.599525 22719 raft_consensus.cc:695] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 1 LEADER]: Becoming Leader. State: Replica: 05d0551bc0f04bdd8fad024b9196fc0d, State: Running, Role: LEADER
I20250901 20:02:21.599745 22719 consensus_queue.cc:237] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } } peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } }
I20250901 20:02:21.600147 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.599938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.66:38731)
0901 20:02:21.600039 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.600089 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20250901 20:02:21.601939 22150 catalog_manager.cc:5582] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 05d0551bc0f04bdd8fad024b9196fc0d (127.14.67.66). New cstate: current_term: 1 leader_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d0551bc0f04bdd8fad024b9196fc0d" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 } health_report { overall_health: UNKNOWN } } }
I20250901 20:02:21.627311 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.624189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.14.67.66:35033)
0901 20:02:21.624332 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:21.624335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:21.624349 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:02:21.624437 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:21.624440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:21.624498 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:21.624589 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:02:21.624985 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.625469 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:21.626242 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:21.626369 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:21.626423 (+ 54us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:02:21.626880 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:02:21.626978 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:21.627165 (+ 187us) server_negotiation.cc:299] Negotiation successful
0901 20:02:21.627211 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250901 20:02:21.630185 22299 raft_consensus.cc:1273] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4 [term 1 FOLLOWER]: Refusing update from remote peer 05d0551bc0f04bdd8fad024b9196fc0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:21.630475 22571 raft_consensus.cc:1273] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea [term 1 FOLLOWER]: Refusing update from remote peer 05d0551bc0f04bdd8fad024b9196fc0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:21.630786 22719 consensus_queue.cc:1035] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d65920bf7264cb5b463e297265d12c4" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:21.631217 22724 consensus_queue.cc:1035] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "39db649d1bd845a0ac3324d3361819ea" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:21.632486 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.632242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.65:49219)
0901 20:02:21.632355 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.632409 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20250901 20:02:21.634392 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.634167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.67:38411)
0901 20:02:21.634289 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.634335 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20250901 20:02:21.636581 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.636348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.66:58015)
0901 20:02:21.636464 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.636512 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20250901 20:02:21.637068 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.636825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.65:49181)
0901 20:02:21.636953 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.637007 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20250901 20:02:21.637969 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.637719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.66:39027)
0901 20:02:21.637861 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.637905 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20250901 20:02:21.638819 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.638574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.14.67.67:43155)
0901 20:02:21.638710 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.638758 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20250901 20:02:21.638828 22730 mvcc.cc:204] Tried to move back new op lower bound from 7195676432912535552 to 7195676432795066368. Current Snapshot: MvccSnapshot[applied={T|T < 7195676432912535552}]
I20250901 20:02:21.643865 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.643622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36774)
0901 20:02:21.643739 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.643793 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20250901 20:02:21.644569 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:21.644373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39725 (local address 127.0.0.1:36790)
0901 20:02:21.644473 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.644516 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:39725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20250901 20:02:21.644691 22151 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47640:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:21.644800 22151 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47640:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:21.648420 22151 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1c0d1c7626a0438b866868946ac054e2: Sending DeleteTablet for 3 replicas of tablet 5675f84b310c4ed499457eab360b9e3f
I20250901 20:02:21.648907 22415 tablet_service.cc:1515] Processing DeleteTablet for tablet 5675f84b310c4ed499457eab360b9e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:21 UTC) from {username='oryx', principal='oryx/127.14.67.124@KRBTEST.COM'} at 127.0.0.1:41920
I20250901 20:02:21.648994 22279 tablet_service.cc:1515] Processing DeleteTablet for tablet 5675f84b310c4ed499457eab360b9e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:21 UTC) from {username='oryx', principal='oryx/127.14.67.124@KRBTEST.COM'} at 127.0.0.1:36610
I20250901 20:02:21.649238 22736 tablet_replica.cc:331] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d: stopping tablet replica
I20250901 20:02:21.649281 22551 tablet_service.cc:1515] Processing DeleteTablet for tablet 5675f84b310c4ed499457eab360b9e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:21 UTC) from {username='oryx', principal='oryx/127.14.67.124@KRBTEST.COM'} at 127.0.0.1:39110
I20250901 20:02:21.649353 22736 raft_consensus.cc:2241] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:02:21.649593 22736 raft_consensus.cc:2270] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:21.649768 22737 tablet_replica.cc:331] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4: stopping tablet replica
I20250901 20:02:21.649873 22737 raft_consensus.cc:2241] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:21.649999 22737 raft_consensus.cc:2270] T 5675f84b310c4ed499457eab360b9e3f P 7d65920bf7264cb5b463e297265d12c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:21.650216 22736 ts_tablet_manager.cc:1905] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:21.650700 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 22216
I20250901 20:02:21.653296 22736 ts_tablet_manager.cc:1918] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:21.653360 22736 log.cc:1199] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/5675f84b310c4ed499457eab360b9e3f
I20250901 20:02:21.653694 22736 ts_tablet_manager.cc:1939] T 5675f84b310c4ed499457eab360b9e3f P 05d0551bc0f04bdd8fad024b9196fc0d: Deleting consensus metadata
I20250901 20:02:21.654376 22139 catalog_manager.cc:4928] TS 05d0551bc0f04bdd8fad024b9196fc0d (127.14.67.66:35033): tablet 5675f84b310c4ed499457eab360b9e3f (table test-table [id=66b5b8253d97435c88b8e419faf7fcae]) successfully deleted
I20250901 20:02:21.656261 22738 tablet_replica.cc:331] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea: stopping tablet replica
I20250901 20:02:21.656383 22738 raft_consensus.cc:2241] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:21.656507 22738 raft_consensus.cc:2270] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:21.657179 22738 ts_tablet_manager.cc:1905] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:21.659438 22738 ts_tablet_manager.cc:1918] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:21.659510 22738 log.cc:1199] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/5675f84b310c4ed499457eab360b9e3f
I20250901 20:02:21.659842 22738 ts_tablet_manager.cc:1939] T 5675f84b310c4ed499457eab360b9e3f P 39db649d1bd845a0ac3324d3361819ea: Deleting consensus metadata
W20250901 20:02:21.660125 22711 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:21.659643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:40197 (local address 127.0.0.1:36624)
0901 20:02:21.659786 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:21.659804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:21.659879 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:21.660021 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.65:40197: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":41}
W20250901 20:02:21.660251 22137 catalog_manager.cc:4671] TS 7d65920bf7264cb5b463e297265d12c4 (127.14.67.65:40197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5675f84b310c4ed499457eab360b9e3f: Network error: Client connection negotiation failed: client connection to 127.14.67.65:40197: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 20:02:21.660445 22138 catalog_manager.cc:4928] TS 39db649d1bd845a0ac3324d3361819ea (127.14.67.67:42619): tablet 5675f84b310c4ed499457eab360b9e3f (table test-table [id=66b5b8253d97435c88b8e419faf7fcae]) successfully deleted
I20250901 20:02:21.660527 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 22352
I20250901 20:02:21.665722 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 22488
I20250901 20:02:21.672338 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 21829
I20250901 20:02:21.677795 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 21940
2025-09-01T20:02:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9252 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:02:21 dist-test-slave-2s4d krb5kdc[22741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:02:21 dist-test-slave-2s4d krb5kdc[22741](info): set up 2 sockets
Sep 01 20:02:21 dist-test-slave-2s4d krb5kdc[22741](info): commencing operation
krb5kdc: starting...
W20250901 20:02:23.672542 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.971s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:02:23 dist-test-slave-2s4d krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:02:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:02:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
I20250901 20:02:23.811888 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:34321
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:34321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:23.912045 22757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:23.912231 22757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:23.912281 22757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:23.915221 22757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:23.915268 22757 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250901 20:02:23.915287 22757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:23.915306 22757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:23.915324 22757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:23.915341 22757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:02:23.919265 22757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:34321
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:34321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:23.920104 22757 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:23.920804 22757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:23.925873 22765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:23.925935 22763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:23.925915 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:23.926172 22757 server_base.cc:1047] running on GCE node
I20250901 20:02:23.926594 22757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:23.927362 22757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:23.928519 22757 hybrid_clock.cc:648] HybridClock initialized: now 1756756943928505 us; error 31 us; skew 500 ppm
Sep 01 20:02:23 dist-test-slave-2s4d krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756943, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:23.930966 22757 init.cc:377] Logged in from keytab as oryx/127.14.67.126@KRBTEST.COM (short username oryx)
I20250901 20:02:23.931879 22757 webserver.cc:480] Webserver started at http://127.14.67.126:43929/ using document root <none> and password file <none>
I20250901 20:02:23.932365 22757 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:23.932420 22757 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:23.932580 22757 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:23.934123 22757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "b802f41ef7be4bda914bb8397fd1c558"
format_stamp: "Formatted at 2025-09-01 20:02:23 on dist-test-slave-2s4d"
server_key: "a5fb0a982ff61d0ab08d41412c4990d3"
server_key_iv: "726a69b833cecf5c8fc4d51e8d0b0111"
server_key_version: "encryptionkey@0"
I20250901 20:02:23.934615 22757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "b802f41ef7be4bda914bb8397fd1c558"
format_stamp: "Formatted at 2025-09-01 20:02:23 on dist-test-slave-2s4d"
server_key: "a5fb0a982ff61d0ab08d41412c4990d3"
server_key_iv: "726a69b833cecf5c8fc4d51e8d0b0111"
server_key_version: "encryptionkey@0"
I20250901 20:02:23.937491 22757 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250901 20:02:23.939415 22772 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:23.940331 22757 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:02:23.940456 22757 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "b802f41ef7be4bda914bb8397fd1c558"
format_stamp: "Formatted at 2025-09-01 20:02:23 on dist-test-slave-2s4d"
server_key: "a5fb0a982ff61d0ab08d41412c4990d3"
server_key_iv: "726a69b833cecf5c8fc4d51e8d0b0111"
server_key_version: "encryptionkey@0"
I20250901 20:02:23.940543 22757 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:23.953066 22757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:23.956110 22757 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:23.956279 22757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:23.963551 22834 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:34321 every 8 connection(s)
I20250901 20:02:23.963552 22757 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:34321
I20250901 20:02:23.964463 22757 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:02:23.966754 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 22757
I20250901 20:02:23.966799 22835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:23.966881 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:02:23.967123 14605 external_mini_cluster.cc:1468] Setting key 8fd120b205dc37209aa76b6b0663baf9
I20250901 20:02:23.972664 22835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558: Bootstrap starting.
Sep 01 20:02:23 dist-test-slave-2s4d krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:23.974820 22835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:23.975531 22835 log.cc:826] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:23.977445 22835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558: No bootstrap required, opened a new log
I20250901 20:02:23.980427 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:23.968177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.14.67.126:34321)
0901 20:02:23.968977 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:23.968988 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:23.969026 (+ 38us) server_negotiation.cc:407] Connection header received
0901 20:02:23.969579 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:23.969600 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:23.969930 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:23.970296 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:23.971075 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:23.972266 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:23.972929 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:23.973182 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:23.975295 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:23.975312 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:23.975324 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:23.975365 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:23.977327 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:23.978116 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:23.978121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:23.978126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:23.978205 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:23.978479 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:23.978482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:23.978484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:23.978870 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:23.979054 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:23.979449 (+ 395us) server_negotiation.cc:299] Negotiation successful
0901 20:02:23.979684 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":513,"threads_started":1}
I20250901 20:02:23.980496 22835 raft_consensus.cc:357] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b802f41ef7be4bda914bb8397fd1c558" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 34321 } }
I20250901 20:02:23.980669 22835 raft_consensus.cc:383] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:23.980736 22835 raft_consensus.cc:738] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b802f41ef7be4bda914bb8397fd1c558, State: Initialized, Role: FOLLOWER
I20250901 20:02:23.981133 22835 consensus_queue.cc:260] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b802f41ef7be4bda914bb8397fd1c558" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 34321 } }
I20250901 20:02:23.981256 22835 raft_consensus.cc:397] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:02:23.981324 22835 raft_consensus.cc:491] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:02:23.981438 22835 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:23.982312 22835 raft_consensus.cc:513] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b802f41ef7be4bda914bb8397fd1c558" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 34321 } }
I20250901 20:02:23.982659 22835 leader_election.cc:304] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b802f41ef7be4bda914bb8397fd1c558; no voters:
I20250901 20:02:23.982918 22835 leader_election.cc:290] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:02:23.983125 22840 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:23.983348 22840 raft_consensus.cc:695] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [term 1 LEADER]: Becoming Leader. State: Replica: b802f41ef7be4bda914bb8397fd1c558, State: Running, Role: LEADER
I20250901 20:02:23.983600 22840 consensus_queue.cc:237] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b802f41ef7be4bda914bb8397fd1c558" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 34321 } }
I20250901 20:02:23.983924 22835 sys_catalog.cc:564] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:23.985080 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b802f41ef7be4bda914bb8397fd1c558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b802f41ef7be4bda914bb8397fd1c558" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 34321 } } }
I20250901 20:02:23.985217 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:23.985865 22849 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:02:23.986378 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b802f41ef7be4bda914bb8397fd1c558. Latest consensus state: current_term: 1 leader_uuid: "b802f41ef7be4bda914bb8397fd1c558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b802f41ef7be4bda914bb8397fd1c558" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 34321 } } }
I20250901 20:02:23.986462 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:23.988272 22849 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:02:23.993111 22849 catalog_manager.cc:1349] Generated new cluster ID: f595db85610d47c48c75b7e22daead4c
I20250901 20:02:23.993166 22849 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:02:24.019937 22849 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:02:24.020967 22849 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:02:24.031572 22849 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558: Generated new TSK 0
I20250901 20:02:24.032347 22849 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
I20250901 20:02:24.097837 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:34321
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:24.197428 22863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:24.197616 22863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:24.197707 22863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:24.200727 22863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:24.200788 22863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:24.200875 22863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250901 20:02:24.200958 22863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:02:24.204811 22863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.67.126:34321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:24.205753 22863 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:24.206769 22863 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:24.212663 22871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:24.212678 22869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:24.212670 22868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:24.212910 22863 server_base.cc:1047] running on GCE node
I20250901 20:02:24.213231 22863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:24.213698 22863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:24.214962 22863 hybrid_clock.cc:648] HybridClock initialized: now 1756756944214924 us; error 28 us; skew 500 ppm
Sep 01 20:02:24 dist-test-slave-2s4d krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:24.218003 22863 init.cc:377] Logged in from keytab as oryx/127.14.67.65@KRBTEST.COM (short username oryx)
I20250901 20:02:24.219026 22863 webserver.cc:480] Webserver started at http://127.14.67.65:34823/ using document root <none> and password file <none>
I20250901 20:02:24.219502 22863 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:24.219548 22863 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:24.219687 22863 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:24.221222 22863 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "cb295f74a05d4d3fa67899151432b65b"
format_stamp: "Formatted at 2025-09-01 20:02:24 on dist-test-slave-2s4d"
server_key: "38577500138aa0a10e9c5ef7244f9b7f"
server_key_iv: "09247b5d5ff574c471b1e4dfb0f61f1f"
server_key_version: "encryptionkey@0"
I20250901 20:02:24.221626 22863 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "cb295f74a05d4d3fa67899151432b65b"
format_stamp: "Formatted at 2025-09-01 20:02:24 on dist-test-slave-2s4d"
server_key: "38577500138aa0a10e9c5ef7244f9b7f"
server_key_iv: "09247b5d5ff574c471b1e4dfb0f61f1f"
server_key_version: "encryptionkey@0"
I20250901 20:02:24.224712 22863 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250901 20:02:24.227181 22878 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:24.228511 22863 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250901 20:02:24.228670 22863 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "cb295f74a05d4d3fa67899151432b65b"
format_stamp: "Formatted at 2025-09-01 20:02:24 on dist-test-slave-2s4d"
server_key: "38577500138aa0a10e9c5ef7244f9b7f"
server_key_iv: "09247b5d5ff574c471b1e4dfb0f61f1f"
server_key_version: "encryptionkey@0"
I20250901 20:02:24.228808 22863 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:24.249562 22863 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:24.253119 22863 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:24.253302 22863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:24.254716 22863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:24.254767 22863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:24.254805 22863 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:24.254822 22863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 01 20:02:24 dist-test-slave-2s4d krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:24.265995 22863 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:33435
I20250901 20:02:24.266017 22993 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:33435 every 8 connection(s)
I20250901 20:02:24.266858 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.255762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:57421 (local address 127.14.67.126:34321)
0901 20:02:24.255883 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.255887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.256670 (+ 783us) server_negotiation.cc:407] Connection header received
0901 20:02:24.257560 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.257563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.257626 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.257682 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:24.259175 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.259701 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.260302 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.260475 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.263099 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:24.263121 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:24.263124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:24.263157 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:24.264973 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.265477 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.265481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.265483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.265524 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.265810 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.265813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.265814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.265960 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:24.266049 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.266574 (+ 525us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.266695 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250901 20:02:24.267392 22863 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:02:24.267436 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.256086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:34321 (local address 127.14.67.65:57421)
0901 20:02:24.256527 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:24.256561 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:24.257344 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:24.258039 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:24.258049 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:24.258423 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:24.259025 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.259042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.259807 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.259811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:24.260197 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.260204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.260379 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.261016 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:24.261054 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:24.262940 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:24.265127 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:24.265131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:24.265138 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:24.265353 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:24.265613 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:24.265616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:24.265617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:24.265694 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:24.266064 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:24.266072 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:24.266329 (+ 257us) client_negotiation.cc:769] Sending connection context
0901 20:02:24.266527 (+ 198us) client_negotiation.cc:240] Negotiation successful
0901 20:02:24.266786 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20250901 20:02:24.270828 22994 heartbeater.cc:344] Connected to a master server at 127.14.67.126:34321
I20250901 20:02:24.271010 22994 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:24.271796 22994 heartbeater.cc:507] Master 127.14.67.126:34321 requested a full tablet report, sending...
I20250901 20:02:24.273450 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 22863
I20250901 20:02:24.273442 22789 ts_manager.cc:194] Registered new tserver with Master: cb295f74a05d4d3fa67899151432b65b (127.14.67.65:33435)
I20250901 20:02:24.273558 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:02:24.273808 14605 external_mini_cluster.cc:1468] Setting key 127d5f2a39a08a8b24b674dd0e65b155
I20250901 20:02:24.274911 22789 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:57421
I20250901 20:02:24.278326 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.271139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:44649 (local address 127.14.67.126:34321)
0901 20:02:24.271271 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.271275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.271489 (+ 214us) server_negotiation.cc:407] Connection header received
0901 20:02:24.271694 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.271696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.271763 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.271832 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:24.272540 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.273365 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.274373 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.274908 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.275341 (+ 433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:24.275358 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:24.275361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:24.275386 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:24.276880 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.277347 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.277351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.277353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.277398 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.277624 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.277628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.277629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.277756 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:24.277845 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.278086 (+ 241us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.278202 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20250901 20:02:24.278331 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.271008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:34321 (local address 127.14.67.65:44649)
0901 20:02:24.271383 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:24.271400 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:24.271543 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:24.271890 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:24.271893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:24.272128 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:24.272413 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.272421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.273536 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.273541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:24.274134 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.274145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.274265 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.274816 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:24.274832 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:24.275203 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:24.276987 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:24.276993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:24.276996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:24.277230 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:24.277493 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:24.277495 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:24.277497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:24.277537 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:24.277845 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:24.277849 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:24.277938 (+ 89us) client_negotiation.cc:769] Sending connection context
0901 20:02:24.278029 (+ 91us) client_negotiation.cc:240] Negotiation successful
0901 20:02:24.278155 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for oryx/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
I20250901 20:02:24.325503 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:34321
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:24.424441 23005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:24.424628 23005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:24.424714 23005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:24.427726 23005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:24.427784 23005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:24.427862 23005 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250901 20:02:24.427933 23005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:02:24.431834 23005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.67.126:34321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:24.432703 23005 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:24.433470 23005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:24.439284 23013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:24.439340 23010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:24.439294 23011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:24.439535 23005 server_base.cc:1047] running on GCE node
I20250901 20:02:24.439841 23005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:24.440318 23005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:24.441555 23005 hybrid_clock.cc:648] HybridClock initialized: now 1756756944441530 us; error 31 us; skew 500 ppm
Sep 01 20:02:24 dist-test-slave-2s4d krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:24.445075 23005 init.cc:377] Logged in from keytab as oryx/127.14.67.66@KRBTEST.COM (short username oryx)
I20250901 20:02:24.446348 23005 webserver.cc:480] Webserver started at http://127.14.67.66:37153/ using document root <none> and password file <none>
I20250901 20:02:24.447006 23005 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:24.447067 23005 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:24.447274 23005 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:24.449226 23005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "a9ace1800c014ff18b217adfb028f7a4"
format_stamp: "Formatted at 2025-09-01 20:02:24 on dist-test-slave-2s4d"
server_key: "25e871053d49bd18061e083be4fc913f"
server_key_iv: "db7d9af5efc7e66891bbb3400be77aa9"
server_key_version: "encryptionkey@0"
I20250901 20:02:24.449739 23005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "a9ace1800c014ff18b217adfb028f7a4"
format_stamp: "Formatted at 2025-09-01 20:02:24 on dist-test-slave-2s4d"
server_key: "25e871053d49bd18061e083be4fc913f"
server_key_iv: "db7d9af5efc7e66891bbb3400be77aa9"
server_key_version: "encryptionkey@0"
I20250901 20:02:24.453578 23005 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250901 20:02:24.455945 23020 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:24.456959 23005 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:02:24.457073 23005 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "a9ace1800c014ff18b217adfb028f7a4"
format_stamp: "Formatted at 2025-09-01 20:02:24 on dist-test-slave-2s4d"
server_key: "25e871053d49bd18061e083be4fc913f"
server_key_iv: "db7d9af5efc7e66891bbb3400be77aa9"
server_key_version: "encryptionkey@0"
I20250901 20:02:24.457186 23005 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:24.478909 23005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:24.481714 23005 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:24.481889 23005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:24.483318 23005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:24.483367 23005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:24.483423 23005 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:24.483457 23005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 01 20:02:24 dist-test-slave-2s4d krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:24.493963 23005 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:40941
I20250901 20:02:24.494093 23135 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:40941 every 8 connection(s)
I20250901 20:02:24.494278 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.483896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:35161 (local address 127.14.67.126:34321)
0901 20:02:24.484033 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.484038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.484683 (+ 645us) server_negotiation.cc:407] Connection header received
0901 20:02:24.485683 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.485686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.485748 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.485822 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:24.487208 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.487903 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.488511 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.488678 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.490947 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:24.490963 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:24.490965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:24.490987 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:24.492416 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.492848 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.492855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.492860 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.492918 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.493225 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.493228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.493230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.493419 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:24.493532 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.494018 (+ 486us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.494132 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250901 20:02:24.494998 23005 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:02:24.495398 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.484128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:34321 (local address 127.14.67.66:35161)
0901 20:02:24.484558 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:24.484597 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:24.485486 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:24.485984 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:24.485994 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:24.486468 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:24.487065 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.487078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.488015 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.488018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:24.488401 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.488408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.488585 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.489176 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:24.489203 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:24.490805 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:24.492518 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:24.492523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:24.492532 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:24.492747 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:24.493048 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:24.493050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:24.493052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:24.493129 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:24.493537 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:24.493545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:24.493786 (+ 241us) client_negotiation.cc:769] Sending connection context
0901 20:02:24.493970 (+ 184us) client_negotiation.cc:240] Negotiation successful
0901 20:02:24.494233 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":94,"threads_started":1}
I20250901 20:02:24.497268 23136 heartbeater.cc:344] Connected to a master server at 127.14.67.126:34321
I20250901 20:02:24.497427 23136 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:24.497910 23136 heartbeater.cc:507] Master 127.14.67.126:34321 requested a full tablet report, sending...
I20250901 20:02:24.498847 22789 ts_manager.cc:194] Registered new tserver with Master: a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941)
I20250901 20:02:24.499433 22789 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:35161
I20250901 20:02:24.501355 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 23005
I20250901 20:02:24.501713 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:02:24.501974 14605 external_mini_cluster.cc:1468] Setting key 0fc25b2f176397322c342211ced6bb15
I20250901 20:02:24.505954 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.499218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:37669 (local address 127.14.67.126:34321)
0901 20:02:24.499343 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.499346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.499537 (+ 191us) server_negotiation.cc:407] Connection header received
0901 20:02:24.499657 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.499660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.499701 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.499763 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:24.500253 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.500696 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.501277 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.501481 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.502182 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:24.502199 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:24.502202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:24.502223 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:24.504184 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.504673 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.504679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.504683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.504754 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.504971 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.504975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.504979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.505247 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:24.505349 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.505638 (+ 289us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.505776 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250901 20:02:24.505976 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.499150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:34321 (local address 127.14.67.66:37669)
0901 20:02:24.499455 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:24.499466 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:24.499565 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:24.499787 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:24.499790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:24.499948 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:24.500137 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.500143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.500799 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.500802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:24.501162 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.501168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.501285 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.501744 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:24.501756 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:24.502033 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:24.504356 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:24.504362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:24.504367 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:24.504560 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:24.504847 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:24.504849 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:24.504851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:24.504889 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:24.505358 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:24.505365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:24.505469 (+ 104us) client_negotiation.cc:769] Sending connection context
0901 20:02:24.505578 (+ 109us) client_negotiation.cc:240] Negotiation successful
0901 20:02:24.505769 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for oryx/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab.
I20250901 20:02:24.545699 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:34321
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:24.645033 23147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:24.645303 23147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:24.645409 23147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:24.648454 23147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:24.648512 23147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:24.648599 23147 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250901 20:02:24.648676 23147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:02:24.652551 23147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/krb5kdc/oryx.keytab
--principal=oryx/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.67.126:34321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:24.653434 23147 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:24.654179 23147 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:24.659992 23152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:24.660002 23153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:24.660007 23155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:24.660053 23147 server_base.cc:1047] running on GCE node
I20250901 20:02:24.660605 23147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:24.661120 23147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:24.662297 23147 hybrid_clock.cc:648] HybridClock initialized: now 1756756944662282 us; error 28 us; skew 500 ppm
Sep 01 20:02:24 dist-test-slave-2s4d krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:24.665609 23147 init.cc:377] Logged in from keytab as oryx/127.14.67.67@KRBTEST.COM (short username oryx)
I20250901 20:02:24.666913 23147 webserver.cc:480] Webserver started at http://127.14.67.67:35175/ using document root <none> and password file <none>
I20250901 20:02:24.667515 23147 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:24.667565 23147 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:24.667745 23147 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:24.669623 23147 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "c2634c0c7fa74861af82abdc0a076d97"
format_stamp: "Formatted at 2025-09-01 20:02:24 on dist-test-slave-2s4d"
server_key: "2735bcab23a41e04a8aff468cb9470df"
server_key_iv: "c29b290f182da92101e9207a872d4efb"
server_key_version: "encryptionkey@0"
I20250901 20:02:24.670120 23147 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "c2634c0c7fa74861af82abdc0a076d97"
format_stamp: "Formatted at 2025-09-01 20:02:24 on dist-test-slave-2s4d"
server_key: "2735bcab23a41e04a8aff468cb9470df"
server_key_iv: "c29b290f182da92101e9207a872d4efb"
server_key_version: "encryptionkey@0"
I20250901 20:02:24.673722 23147 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250901 20:02:24.676119 23162 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:24.677027 23147 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250901 20:02:24.677114 23147 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "c2634c0c7fa74861af82abdc0a076d97"
format_stamp: "Formatted at 2025-09-01 20:02:24 on dist-test-slave-2s4d"
server_key: "2735bcab23a41e04a8aff468cb9470df"
server_key_iv: "c29b290f182da92101e9207a872d4efb"
server_key_version: "encryptionkey@0"
I20250901 20:02:24.677196 23147 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:24.688865 23147 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:24.691639 23147 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:24.691789 23147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:24.693202 23147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:24.693266 23147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:24.693317 23147 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:24.693342 23147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 01 20:02:24 dist-test-slave-2s4d krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:24.703980 23147 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:34857
I20250901 20:02:24.704229 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.693820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:51101 (local address 127.14.67.126:34321)
0901 20:02:24.693962 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.693967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.694713 (+ 746us) server_negotiation.cc:407] Connection header received
0901 20:02:24.695551 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.695555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.695623 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.695721 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:24.696970 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.697484 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.698101 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.698270 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.700492 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:24.700512 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:24.700515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:24.700543 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:24.702340 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.702899 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.702905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.702908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.702963 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.703228 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.703231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.703233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.703408 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:24.703516 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.703953 (+ 437us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.704064 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250901 20:02:24.704780 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.694055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:34321 (local address 127.14.67.67:51101)
0901 20:02:24.694468 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:24.694532 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:24.695368 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:24.695880 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:24.695891 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:24.696282 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:24.696823 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.696835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.697592 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.697596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:24.697989 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.697999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.698201 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.698892 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:24.698919 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:24.700357 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:24.702455 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:24.702460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:24.702469 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:24.702795 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:24.703057 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:24.703060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:24.703061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:24.703135 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:24.703499 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:24.703503 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:24.703685 (+ 182us) client_negotiation.cc:769] Sending connection context
0901 20:02:24.703886 (+ 201us) client_negotiation.cc:240] Negotiation successful
0901 20:02:24.704117 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20250901 20:02:24.704898 23147 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:02:24.705466 23277 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:34857 every 8 connection(s)
I20250901 20:02:24.706395 23278 heartbeater.cc:344] Connected to a master server at 127.14.67.126:34321
I20250901 20:02:24.706574 23278 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:24.707131 23278 heartbeater.cc:507] Master 127.14.67.126:34321 requested a full tablet report, sending...
I20250901 20:02:24.708029 22789 ts_manager.cc:194] Registered new tserver with Master: c2634c0c7fa74861af82abdc0a076d97 (127.14.67.67:34857)
I20250901 20:02:24.708684 22789 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:51101
I20250901 20:02:24.711930 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 23147
I20250901 20:02:24.712163 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:02:24.712499 14605 external_mini_cluster.cc:1468] Setting key 0d1f9681098e342e8285de42e1be5af5
I20250901 20:02:24.714643 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250901 20:02:24.715354 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.707907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:34321 (local address 127.14.67.67:35889)
0901 20:02:24.708203 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:24.708215 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:24.708315 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:24.708563 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:24.708566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:24.708710 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:24.708923 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.708929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.709925 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.709930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:24.710406 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.710418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.710646 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.711279 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:24.711294 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:24.711608 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:24.713705 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:24.713709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:24.713711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:24.713962 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:24.714255 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:24.714260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:24.714262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:24.714330 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:24.714778 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:24.714781 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:24.714866 (+ 85us) client_negotiation.cc:769] Sending connection context
0901 20:02:24.714995 (+ 129us) client_negotiation.cc:240] Negotiation successful
0901 20:02:24.715152 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20250901 20:02:24.715611 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.707990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:35889 (local address 127.14.67.126:34321)
0901 20:02:24.708111 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.708114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.708280 (+ 166us) server_negotiation.cc:407] Connection header received
0901 20:02:24.708412 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.708416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.708458 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.708533 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:24.709029 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.709704 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.710632 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.710756 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.711787 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:24.711800 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:24.711802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:24.711823 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:24.713585 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.714082 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.714085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.714086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.714127 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.714434 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.714437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.714438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.714636 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:24.714732 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.715109 (+ 377us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.715325 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 01 20:02:24 dist-test-slave-2s4d krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756944, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 01 20:02:24 dist-test-slave-2s4d krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:24 dist-test-slave-2s4d krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
W20250901 20:02:24.736508 22838 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:24.730259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.14.67.126:34321)
0901 20:02:24.730397 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.730400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.730411 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:02:24.730443 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.730445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.730480 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.730569 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:24.731323 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.731804 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.732428 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.732630 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.736384 (+ 3754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38316: BlockingRecv error: recv got EOF from 127.0.0.1:38316 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 01 20:02:24 dist-test-slave-2s4d krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756944, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:24.744956 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.738140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.14.67.126:34321)
0901 20:02:24.738238 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.738241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.738322 (+ 81us) server_negotiation.cc:407] Connection header received
0901 20:02:24.738402 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.738405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.738437 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.738528 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:24.739132 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.739572 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.740108 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.740220 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.742196 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:24.742213 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:24.742216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:24.742238 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:24.743563 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.744092 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.744095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.744096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.744135 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:24.744353 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:24.744355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:24.744356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:24.744482 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:24.744581 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.744780 (+ 199us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.744856 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20250901 20:02:24.747951 22789 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 20:02:24.750782 22789 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:02:24.763407 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.759066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:40941 (local address 127.0.0.1:43328)
0901 20:02:24.759690 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:24.759704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:24.759789 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:24.760427 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:24.760432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:24.760474 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:24.760731 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.760738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.762145 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.762148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:24.762999 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.763006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.763087 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.763098 (+ 11us) client_negotiation.cc:769] Sending connection context
0901 20:02:24.763172 (+ 74us) client_negotiation.cc:240] Negotiation successful
0901 20:02:24.763254 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"spinlock_wait_cycles":7808,"thread_start_us":90,"threads_started":1}
I20250901 20:02:24.763407 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.758938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:34857 (local address 127.0.0.1:32940)
0901 20:02:24.759267 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:24.759302 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:24.759419 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:24.760663 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:24.760666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:24.760682 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:24.760889 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.760899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.761994 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.761997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:24.762900 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.762909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.763030 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.763087 (+ 57us) client_negotiation.cc:769] Sending connection context
0901 20:02:24.763172 (+ 85us) client_negotiation.cc:240] Negotiation successful
0901 20:02:24.763254 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
I20250901 20:02:24.763948 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.759114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.14.67.67:34857)
0901 20:02:24.760278 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.760285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.760306 (+ 21us) server_negotiation.cc:407] Connection header received
0901 20:02:24.760381 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.760386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.760537 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.760667 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:24.761012 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.761864 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.763059 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.763556 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.763670 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.763726 (+ 56us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.763793 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1056,"thread_start_us":104,"threads_started":1}
I20250901 20:02:24.764442 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.759538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.14.67.66:40941)
0901 20:02:24.759973 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.759980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.760002 (+ 22us) server_negotiation.cc:407] Connection header received
0901 20:02:24.760085 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.760090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.760274 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.760413 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:24.760841 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.762024 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.763744 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.764138 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.764231 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.764280 (+ 49us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.764334 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":98,"threads_started":1}
I20250901 20:02:24.766340 23210 tablet_service.cc:1468] Processing CreateTablet for tablet 8b928f914ef94163988e49bb48b0e3ad (DEFAULT_TABLE table=test-table [id=b587ee48833b44cb9233202338a97f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:24.766341 23070 tablet_service.cc:1468] Processing CreateTablet for tablet 8b928f914ef94163988e49bb48b0e3ad (DEFAULT_TABLE table=test-table [id=b587ee48833b44cb9233202338a97f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:24.767076 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.759902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:33435 (local address 127.0.0.1:33360)
0901 20:02:24.760367 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:24.760379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:24.760474 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:24.760916 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:24.760921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:24.760952 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:24.761194 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.761202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.762486 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.762490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:24.763356 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.763366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.763471 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.766635 (+ 3164us) client_negotiation.cc:769] Sending connection context
0901 20:02:24.766788 (+ 153us) client_negotiation.cc:240] Negotiation successful
0901 20:02:24.766901 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"spinlock_wait_cycles":4352,"thread_start_us":103,"threads_started":1}
I20250901 20:02:24.767311 23210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b928f914ef94163988e49bb48b0e3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:24.767311 23070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b928f914ef94163988e49bb48b0e3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:24.767592 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.760063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33360 (local address 127.14.67.65:33435)
0901 20:02:24.760319 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.760329 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.760561 (+ 232us) server_negotiation.cc:407] Connection header received
0901 20:02:24.760644 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.760648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.760786 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.760929 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:24.761304 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.762371 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.763973 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.764470 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.764553 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.767073 (+ 2520us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.767469 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
I20250901 20:02:24.770172 22928 tablet_service.cc:1468] Processing CreateTablet for tablet 8b928f914ef94163988e49bb48b0e3ad (DEFAULT_TABLE table=test-table [id=b587ee48833b44cb9233202338a97f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:24.771144 22928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b928f914ef94163988e49bb48b0e3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:24.773452 23307 tablet_bootstrap.cc:492] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97: Bootstrap starting.
I20250901 20:02:24.773823 23306 tablet_bootstrap.cc:492] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4: Bootstrap starting.
I20250901 20:02:24.776516 23307 tablet_bootstrap.cc:654] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:24.777159 23308 tablet_bootstrap.cc:492] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b: Bootstrap starting.
I20250901 20:02:24.777899 23307 log.cc:826] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:24.778049 23306 tablet_bootstrap.cc:654] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:24.779177 23308 tablet_bootstrap.cc:654] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:24.779207 23306 log.cc:826] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:24.779459 23307 tablet_bootstrap.cc:492] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97: No bootstrap required, opened a new log
I20250901 20:02:24.779624 23307 ts_tablet_manager.cc:1397] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250901 20:02:24.780400 23308 log.cc:826] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:24.780855 23306 tablet_bootstrap.cc:492] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4: No bootstrap required, opened a new log
I20250901 20:02:24.781026 23306 ts_tablet_manager.cc:1397] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250901 20:02:24.782135 23308 tablet_bootstrap.cc:492] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b: No bootstrap required, opened a new log
I20250901 20:02:24.782298 23308 ts_tablet_manager.cc:1397] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250901 20:02:24.782536 23307 raft_consensus.cc:357] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } }
I20250901 20:02:24.782732 23307 raft_consensus.cc:383] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:24.782784 23307 raft_consensus.cc:738] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2634c0c7fa74861af82abdc0a076d97, State: Initialized, Role: FOLLOWER
I20250901 20:02:24.783191 23307 consensus_queue.cc:260] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } }
I20250901 20:02:24.783779 23278 heartbeater.cc:499] Master 127.14.67.126:34321 was elected leader, sending a full tablet report...
I20250901 20:02:24.783996 23307 ts_tablet_manager.cc:1428] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250901 20:02:24.783880 23306 raft_consensus.cc:357] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } }
I20250901 20:02:24.784071 23306 raft_consensus.cc:383] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:24.784128 23306 raft_consensus.cc:738] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ace1800c014ff18b217adfb028f7a4, State: Initialized, Role: FOLLOWER
I20250901 20:02:24.784520 23306 consensus_queue.cc:260] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } }
I20250901 20:02:24.785308 23306 ts_tablet_manager.cc:1428] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250901 20:02:24.785192 23308 raft_consensus.cc:357] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } }
I20250901 20:02:24.785395 23308 raft_consensus.cc:383] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:24.785451 23308 raft_consensus.cc:738] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb295f74a05d4d3fa67899151432b65b, State: Initialized, Role: FOLLOWER
I20250901 20:02:24.785602 23136 heartbeater.cc:499] Master 127.14.67.126:34321 was elected leader, sending a full tablet report...
I20250901 20:02:24.785844 23308 consensus_queue.cc:260] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } }
I20250901 20:02:24.786352 22994 heartbeater.cc:499] Master 127.14.67.126:34321 was elected leader, sending a full tablet report...
I20250901 20:02:24.786710 23308 ts_tablet_manager.cc:1428] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250901 20:02:24.907657 23312 raft_consensus.cc:491] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:02:24.907976 23312 raft_consensus.cc:513] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } }
I20250901 20:02:24.909348 23312 leader_election.cc:290] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941), cb295f74a05d4d3fa67899151432b65b (127.14.67.65:33435)
I20250901 20:02:24.912580 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.909608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:40941 (local address 127.14.67.67:45969)
0901 20:02:24.909781 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:24.909799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:24.909907 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:24.910247 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:24.910250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:24.910269 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:24.910563 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.910570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.911560 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.911563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:24.912165 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.912173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.912298 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.912321 (+ 23us) client_negotiation.cc:769] Sending connection context
0901 20:02:24.912376 (+ 55us) client_negotiation.cc:240] Negotiation successful
0901 20:02:24.912436 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"spinlock_wait_cycles":3456}
I20250901 20:02:24.913048 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.909845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:45969 (local address 127.14.67.66:40941)
0901 20:02:24.910021 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.910026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.910042 (+ 16us) server_negotiation.cc:407] Connection header received
0901 20:02:24.910095 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.910097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.910142 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.910215 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:24.910687 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.911378 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.912301 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.912777 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.912825 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.912906 (+ 81us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.912962 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20250901 20:02:24.913585 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.909829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:58825 (local address 127.14.67.65:33435)
0901 20:02:24.909975 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.909981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.910260 (+ 279us) server_negotiation.cc:407] Connection header received
0901 20:02:24.910442 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.910445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.910490 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.910587 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:24.911020 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.911511 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.912786 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.913200 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.913243 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.913441 (+ 198us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.913496 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250901 20:02:24.913584 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.909751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:33435 (local address 127.14.67.67:58825)
0901 20:02:24.910162 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:24.910175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:24.910326 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:24.910603 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:24.910606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:24.910625 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:24.910897 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.910904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.912015 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.912018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:24.912581 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:24.912588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.913302 (+ 714us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.913315 (+ 13us) client_negotiation.cc:769] Sending connection context
0901 20:02:24.913393 (+ 78us) client_negotiation.cc:240] Negotiation successful
0901 20:02:24.913449 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"spinlock_wait_cycles":4224,"thread_start_us":136,"threads_started":1}
I20250901 20:02:24.913679 23090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b928f914ef94163988e49bb48b0e3ad" candidate_uuid: "c2634c0c7fa74861af82abdc0a076d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ace1800c014ff18b217adfb028f7a4" is_pre_election: true
I20250901 20:02:24.914036 23090 raft_consensus.cc:2466] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2634c0c7fa74861af82abdc0a076d97 in term 0.
I20250901 20:02:24.914222 22948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b928f914ef94163988e49bb48b0e3ad" candidate_uuid: "c2634c0c7fa74861af82abdc0a076d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb295f74a05d4d3fa67899151432b65b" is_pre_election: true
I20250901 20:02:24.914448 22948 raft_consensus.cc:2466] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2634c0c7fa74861af82abdc0a076d97 in term 0.
I20250901 20:02:24.914721 23163 leader_election.cc:304] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ace1800c014ff18b217adfb028f7a4, c2634c0c7fa74861af82abdc0a076d97; no voters:
I20250901 20:02:24.915144 23312 raft_consensus.cc:2802] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:02:24.915226 23312 raft_consensus.cc:491] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:02:24.915287 23312 raft_consensus.cc:3058] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:24.916283 23312 raft_consensus.cc:513] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } }
I20250901 20:02:24.916653 23312 leader_election.cc:290] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [CANDIDATE]: Term 1 election: Requested vote from peers a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941), cb295f74a05d4d3fa67899151432b65b (127.14.67.65:33435)
I20250901 20:02:24.916965 22948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b928f914ef94163988e49bb48b0e3ad" candidate_uuid: "c2634c0c7fa74861af82abdc0a076d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb295f74a05d4d3fa67899151432b65b"
I20250901 20:02:24.916977 23090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b928f914ef94163988e49bb48b0e3ad" candidate_uuid: "c2634c0c7fa74861af82abdc0a076d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ace1800c014ff18b217adfb028f7a4"
I20250901 20:02:24.917060 23090 raft_consensus.cc:3058] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:24.917073 22948 raft_consensus.cc:3058] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:24.917938 22948 raft_consensus.cc:2466] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2634c0c7fa74861af82abdc0a076d97 in term 1.
I20250901 20:02:24.917963 23090 raft_consensus.cc:2466] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2634c0c7fa74861af82abdc0a076d97 in term 1.
I20250901 20:02:24.918251 23164 leader_election.cc:304] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2634c0c7fa74861af82abdc0a076d97, cb295f74a05d4d3fa67899151432b65b; no voters:
I20250901 20:02:24.918467 23312 raft_consensus.cc:2802] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:24.918694 23312 raft_consensus.cc:695] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 1 LEADER]: Becoming Leader. State: Replica: c2634c0c7fa74861af82abdc0a076d97, State: Running, Role: LEADER
I20250901 20:02:24.918977 23312 consensus_queue.cc:237] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } }
I20250901 20:02:24.921998 22788 catalog_manager.cc:5582] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2634c0c7fa74861af82abdc0a076d97 (127.14.67.67). New cstate: current_term: 1 leader_uuid: "c2634c0c7fa74861af82abdc0a076d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } health_report { overall_health: UNKNOWN } } }
I20250901 20:02:24.954069 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:24.950803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.14.67.67:34857)
0901 20:02:24.950942 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:24.950946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:24.950960 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:02:24.951157 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:24.951160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:24.951228 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:24.951329 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:02:24.951812 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.952340 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:24.953071 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:24.953228 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:24.953282 (+ 54us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:02:24.953709 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:02:24.953800 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:24.953910 (+ 110us) server_negotiation.cc:299] Negotiation successful
0901 20:02:24.953958 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20250901 20:02:24.955664 23279 tablet.cc:2378] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 20:02:24.959075 23090 raft_consensus.cc:1273] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Refusing update from remote peer c2634c0c7fa74861af82abdc0a076d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:24.959077 22948 raft_consensus.cc:1273] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Refusing update from remote peer c2634c0c7fa74861af82abdc0a076d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:24.959625 23312 consensus_queue.cc:1035] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:24.959798 23316 consensus_queue.cc:1035] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:24.973019 23321 mvcc.cc:204] Tried to move back new op lower bound from 7195676446546350080 to 7195676446390648832. Current Snapshot: MvccSnapshot[applied={T|T < 7195676446546350080}]
I20250901 20:02:24.975821 22789 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38326:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:24.975984 22789 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38326:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:24.978034 22789 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558: Sending DeleteTablet for 3 replicas of tablet 8b928f914ef94163988e49bb48b0e3ad
I20250901 20:02:24.978667 23210 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b928f914ef94163988e49bb48b0e3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:24 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:32940
I20250901 20:02:24.978746 23070 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b928f914ef94163988e49bb48b0e3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:24 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:43328
I20250901 20:02:24.978866 22928 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b928f914ef94163988e49bb48b0e3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:24 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:33360
I20250901 20:02:24.979112 23329 tablet_replica.cc:331] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97: stopping tablet replica
I20250901 20:02:24.979257 23331 tablet_replica.cc:331] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b: stopping tablet replica
I20250901 20:02:24.979439 23329 raft_consensus.cc:2241] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:02:24.979475 23331 raft_consensus.cc:2241] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:24.979588 22789 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 20:02:24.979745 23331 raft_consensus.cc:2270] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 20:02:24.979993 22789 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:02:24.980016 23329 raft_consensus.cc:2270] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:24.981304 23329 ts_tablet_manager.cc:1905] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:24.981670 23330 tablet_replica.cc:331] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4: stopping tablet replica
I20250901 20:02:24.982036 23331 ts_tablet_manager.cc:1905] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:24.983531 23330 raft_consensus.cc:2241] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:24.983811 23330 raft_consensus.cc:2270] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:24.984440 23329 ts_tablet_manager.cc:1918] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:24.984527 23329 log.cc:1199] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/8b928f914ef94163988e49bb48b0e3ad
I20250901 20:02:24.984633 22928 tablet_service.cc:1468] Processing CreateTablet for tablet aa817b7a1f5542da8f151fc3760d6cd0 (DEFAULT_TABLE table=test-table [id=83d35f38cf0247ecb44908b92caabe9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:24.984838 23329 ts_tablet_manager.cc:1939] T 8b928f914ef94163988e49bb48b0e3ad P c2634c0c7fa74861af82abdc0a076d97: Deleting consensus metadata
I20250901 20:02:24.984911 22928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa817b7a1f5542da8f151fc3760d6cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:24.985144 23210 tablet_service.cc:1468] Processing CreateTablet for tablet aa817b7a1f5542da8f151fc3760d6cd0 (DEFAULT_TABLE table=test-table [id=83d35f38cf0247ecb44908b92caabe9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:24.985370 23210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa817b7a1f5542da8f151fc3760d6cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:24.985868 22776 catalog_manager.cc:4928] TS c2634c0c7fa74861af82abdc0a076d97 (127.14.67.67:34857): tablet 8b928f914ef94163988e49bb48b0e3ad (table test-table [id=b587ee48833b44cb9233202338a97f63]) successfully deleted
I20250901 20:02:24.987105 23307 tablet_bootstrap.cc:492] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97: Bootstrap starting.
I20250901 20:02:24.987283 23070 tablet_service.cc:1468] Processing CreateTablet for tablet aa817b7a1f5542da8f151fc3760d6cd0 (DEFAULT_TABLE table=test-table [id=83d35f38cf0247ecb44908b92caabe9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:24.987526 23070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa817b7a1f5542da8f151fc3760d6cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:24.988103 23307 tablet_bootstrap.cc:654] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:24.989504 23307 tablet_bootstrap.cc:492] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97: No bootstrap required, opened a new log
I20250901 20:02:24.989583 23307 ts_tablet_manager.cc:1397] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250901 20:02:24.989605 23306 tablet_bootstrap.cc:492] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4: Bootstrap starting.
I20250901 20:02:24.990054 23307 raft_consensus.cc:357] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:24.990163 23307 raft_consensus.cc:383] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:24.990200 23307 raft_consensus.cc:738] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2634c0c7fa74861af82abdc0a076d97, State: Initialized, Role: FOLLOWER
I20250901 20:02:24.990315 23307 consensus_queue.cc:260] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:24.990608 23307 ts_tablet_manager.cc:1428] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:02:24.990633 23306 tablet_bootstrap.cc:654] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:24.991642 23330 ts_tablet_manager.cc:1905] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:24.992805 23306 tablet_bootstrap.cc:492] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4: No bootstrap required, opened a new log
I20250901 20:02:24.992895 23306 ts_tablet_manager.cc:1397] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250901 20:02:24.993397 23306 raft_consensus.cc:357] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:24.993516 23306 raft_consensus.cc:383] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:24.993559 23306 raft_consensus.cc:738] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ace1800c014ff18b217adfb028f7a4, State: Initialized, Role: FOLLOWER
I20250901 20:02:24.993706 23306 consensus_queue.cc:260] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:24.993960 23306 ts_tablet_manager.cc:1428] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250901 20:02:24.995155 23331 ts_tablet_manager.cc:1918] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:24.995257 23331 log.cc:1199] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/wal/wals/8b928f914ef94163988e49bb48b0e3ad
I20250901 20:02:24.995546 23331 ts_tablet_manager.cc:1939] T 8b928f914ef94163988e49bb48b0e3ad P cb295f74a05d4d3fa67899151432b65b: Deleting consensus metadata
I20250901 20:02:24.995757 23330 ts_tablet_manager.cc:1918] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:24.995831 23330 log.cc:1199] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/8b928f914ef94163988e49bb48b0e3ad
I20250901 20:02:24.996269 23308 tablet_bootstrap.cc:492] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b: Bootstrap starting.
W20250901 20:02:24.996315 23137 tablet.cc:2378] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 20:02:24.996795 22774 catalog_manager.cc:4928] TS cb295f74a05d4d3fa67899151432b65b (127.14.67.65:33435): tablet 8b928f914ef94163988e49bb48b0e3ad (table test-table [id=b587ee48833b44cb9233202338a97f63]) successfully deleted
I20250901 20:02:24.997170 23308 tablet_bootstrap.cc:654] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:24.996120 23330 ts_tablet_manager.cc:1939] T 8b928f914ef94163988e49bb48b0e3ad P a9ace1800c014ff18b217adfb028f7a4: Deleting consensus metadata
I20250901 20:02:24.998194 23308 tablet_bootstrap.cc:492] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b: No bootstrap required, opened a new log
I20250901 20:02:24.998276 23308 ts_tablet_manager.cc:1397] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250901 20:02:24.998579 22773 catalog_manager.cc:4928] TS a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941): tablet 8b928f914ef94163988e49bb48b0e3ad (table test-table [id=b587ee48833b44cb9233202338a97f63]) successfully deleted
I20250901 20:02:24.998730 23308 raft_consensus.cc:357] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:24.998834 23308 raft_consensus.cc:383] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:24.998867 23308 raft_consensus.cc:738] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb295f74a05d4d3fa67899151432b65b, State: Initialized, Role: FOLLOWER
I20250901 20:02:24.998984 23308 consensus_queue.cc:260] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:24.999233 23308 ts_tablet_manager.cc:1428] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250901 20:02:25.018770 22995 tablet.cc:2378] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250901 20:02:25.076435 23314 raft_consensus.cc:491] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:02:25.076578 23314 raft_consensus.cc:513] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.077642 23314 leader_election.cc:290] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941), c2634c0c7fa74861af82abdc0a076d97 (127.14.67.67:34857)
I20250901 20:02:25.081231 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.077922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:40941 (local address 127.14.67.65:55047)
0901 20:02:25.078286 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:25.078300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:25.078410 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:25.078808 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:25.078810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:25.078822 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:25.079126 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:25.079135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.080136 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.080139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:25.080721 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:25.080728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.080924 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.080948 (+ 24us) client_negotiation.cc:769] Sending connection context
0901 20:02:25.081007 (+ 59us) client_negotiation.cc:240] Negotiation successful
0901 20:02:25.081081 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":76,"threads_started":1}
I20250901 20:02:25.081682 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.078202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:55047 (local address 127.14.67.66:40941)
0901 20:02:25.078329 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:25.078334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:25.078357 (+ 23us) server_negotiation.cc:407] Connection header received
0901 20:02:25.078581 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:25.078585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:25.078634 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:25.078675 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:25.079258 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.080003 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.080886 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.081393 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.081437 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:25.081489 (+ 52us) server_negotiation.cc:299] Negotiation successful
0901 20:02:25.081557 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250901 20:02:25.082067 23090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa817b7a1f5542da8f151fc3760d6cd0" candidate_uuid: "cb295f74a05d4d3fa67899151432b65b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ace1800c014ff18b217adfb028f7a4" is_pre_election: true
I20250901 20:02:25.082219 23090 raft_consensus.cc:2466] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb295f74a05d4d3fa67899151432b65b in term 0.
I20250901 20:02:25.082696 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.077929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:34857 (local address 127.14.67.65:51639)
0901 20:02:25.078364 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:25.078375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:25.078457 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:25.078753 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:25.078758 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:25.078780 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:25.079126 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:25.079135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.081016 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.081020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:25.082300 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:25.082310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.082422 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.082439 (+ 17us) client_negotiation.cc:769] Sending connection context
0901 20:02:25.082486 (+ 47us) client_negotiation.cc:240] Negotiation successful
0901 20:02:25.082563 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":10,"thread_start_us":114,"threads_started":1}
I20250901 20:02:25.083050 22879 leader_election.cc:304] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ace1800c014ff18b217adfb028f7a4, cb295f74a05d4d3fa67899151432b65b; no voters:
I20250901 20:02:25.083405 23314 raft_consensus.cc:2802] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:02:25.083482 23314 raft_consensus.cc:491] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:02:25.083523 23314 raft_consensus.cc:3058] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:25.083990 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.078085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:51639 (local address 127.14.67.67:34857)
0901 20:02:25.078231 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:25.078236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:25.078452 (+ 216us) server_negotiation.cc:407] Connection header received
0901 20:02:25.078582 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:25.078586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:25.078643 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:25.078728 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:25.080142 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.080885 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.083170 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.083711 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.083759 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:25.083813 (+ 54us) server_negotiation.cc:299] Negotiation successful
0901 20:02:25.083876 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20250901 20:02:25.084553 23219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa817b7a1f5542da8f151fc3760d6cd0" candidate_uuid: "cb295f74a05d4d3fa67899151432b65b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2634c0c7fa74861af82abdc0a076d97" is_pre_election: true
I20250901 20:02:25.084650 23314 raft_consensus.cc:513] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.084800 23219 raft_consensus.cc:2466] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb295f74a05d4d3fa67899151432b65b in term 0.
I20250901 20:02:25.085075 23314 leader_election.cc:290] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [CANDIDATE]: Term 1 election: Requested vote from peers a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941), c2634c0c7fa74861af82abdc0a076d97 (127.14.67.67:34857)
I20250901 20:02:25.085482 23090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa817b7a1f5542da8f151fc3760d6cd0" candidate_uuid: "cb295f74a05d4d3fa67899151432b65b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ace1800c014ff18b217adfb028f7a4"
I20250901 20:02:25.085619 23090 raft_consensus.cc:3058] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:25.085726 23219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa817b7a1f5542da8f151fc3760d6cd0" candidate_uuid: "cb295f74a05d4d3fa67899151432b65b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2634c0c7fa74861af82abdc0a076d97"
I20250901 20:02:25.085902 23219 raft_consensus.cc:3058] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:25.086845 23090 raft_consensus.cc:2466] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb295f74a05d4d3fa67899151432b65b in term 1.
I20250901 20:02:25.086989 23219 raft_consensus.cc:2466] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb295f74a05d4d3fa67899151432b65b in term 1.
I20250901 20:02:25.087277 22879 leader_election.cc:304] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ace1800c014ff18b217adfb028f7a4, cb295f74a05d4d3fa67899151432b65b; no voters:
I20250901 20:02:25.087473 23314 raft_consensus.cc:2802] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:25.087672 23314 raft_consensus.cc:695] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 1 LEADER]: Becoming Leader. State: Replica: cb295f74a05d4d3fa67899151432b65b, State: Running, Role: LEADER
I20250901 20:02:25.087911 23314 consensus_queue.cc:237] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.090348 22789 catalog_manager.cc:5582] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b reported cstate change: term changed from 0 to 1, leader changed from <none> to cb295f74a05d4d3fa67899151432b65b (127.14.67.65). New cstate: current_term: 1 leader_uuid: "cb295f74a05d4d3fa67899151432b65b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } health_report { overall_health: UNKNOWN } } }
Sep 01 20:02:25 dist-test-slave-2s4d krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756943, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.126@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:25.150781 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.136122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:34321 (local address 127.0.0.1:38328)
0901 20:02:25.136404 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:25.136418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:25.136513 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:25.136760 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:25.136762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:25.137149 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:25.137334 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:25.137340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.138348 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.138351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:25.138855 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:25.138864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.138975 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.146056 (+ 7081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:25.146071 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:25.147578 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:25.149241 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:25.149249 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:25.149254 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:25.149526 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:25.149747 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:25.149750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:25.149754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:25.149815 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:25.150145 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:25.150155 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:25.150297 (+ 142us) client_negotiation.cc:769] Sending connection context
0901 20:02:25.150427 (+ 130us) client_negotiation.cc:240] Negotiation successful
0901 20:02:25.150603 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":108,"threads_started":1}
I20250901 20:02:25.150784 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.136223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.14.67.126:34321)
0901 20:02:25.136373 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:25.136378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:25.136487 (+ 109us) server_negotiation.cc:407] Connection header received
0901 20:02:25.136595 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:25.136599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:25.136649 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:25.136748 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:25.137451 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.138238 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.138969 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.139146 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.147719 (+ 8573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:25.147736 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:25.147738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:25.147759 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:25.149124 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:25.149628 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:25.149631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:25.149632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:25.149670 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:25.149898 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:25.149900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:25.149901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:25.150046 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:25.150117 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:25.150483 (+ 366us) server_negotiation.cc:299] Negotiation successful
0901 20:02:25.150672 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":54}
I20250901 20:02:25.154472 22789 catalog_manager.cc:2232] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:38328:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250901 20:02:25.155179 22789 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:02:25.159817 22928 tablet_service.cc:1468] Processing CreateTablet for tablet f0a88d6e9aa2460ea790ad6938fd3ba5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9dbb6b89d0f446e3bfa30e3036e7cfa8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250901 20:02:25.159889 23070 tablet_service.cc:1468] Processing CreateTablet for tablet f0a88d6e9aa2460ea790ad6938fd3ba5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9dbb6b89d0f446e3bfa30e3036e7cfa8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250901 20:02:25.159961 23210 tablet_service.cc:1468] Processing CreateTablet for tablet f0a88d6e9aa2460ea790ad6938fd3ba5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9dbb6b89d0f446e3bfa30e3036e7cfa8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250901 20:02:25.160197 22928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0a88d6e9aa2460ea790ad6938fd3ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:25.160230 23070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0a88d6e9aa2460ea790ad6938fd3ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:25.160228 23210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0a88d6e9aa2460ea790ad6938fd3ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:25.162425 23308 tablet_bootstrap.cc:492] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b: Bootstrap starting.
I20250901 20:02:25.162768 23307 tablet_bootstrap.cc:492] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: Bootstrap starting.
I20250901 20:02:25.162957 23306 tablet_bootstrap.cc:492] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4: Bootstrap starting.
I20250901 20:02:25.163458 23308 tablet_bootstrap.cc:654] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:25.163705 23307 tablet_bootstrap.cc:654] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:25.163910 23306 tablet_bootstrap.cc:654] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:25.164744 23308 tablet_bootstrap.cc:492] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b: No bootstrap required, opened a new log
I20250901 20:02:25.164837 23308 ts_tablet_manager.cc:1397] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:25.165042 23306 tablet_bootstrap.cc:492] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4: No bootstrap required, opened a new log
I20250901 20:02:25.165112 23306 ts_tablet_manager.cc:1397] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250901 20:02:25.165480 23308 raft_consensus.cc:357] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.165495 23306 raft_consensus.cc:357] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.165619 23308 raft_consensus.cc:383] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:25.165621 23306 raft_consensus.cc:383] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:25.165665 23306 raft_consensus.cc:738] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ace1800c014ff18b217adfb028f7a4, State: Initialized, Role: FOLLOWER
I20250901 20:02:25.165666 23308 raft_consensus.cc:738] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb295f74a05d4d3fa67899151432b65b, State: Initialized, Role: FOLLOWER
I20250901 20:02:25.165794 23306 consensus_queue.cc:260] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.165797 23308 consensus_queue.cc:260] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.166064 23306 ts_tablet_manager.cc:1428] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:02:25.166019 23313 tablet_replica.cc:440] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } }
I20250901 20:02:25.166102 23308 ts_tablet_manager.cc:1428] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250901 20:02:25.166143 23313 tablet_replica.cc:443] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:02:25.166280 23314 tablet_replica.cc:440] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } }
I20250901 20:02:25.166368 23314 tablet_replica.cc:443] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:02:25.167467 23307 tablet_bootstrap.cc:492] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: No bootstrap required, opened a new log
I20250901 20:02:25.167565 23307 ts_tablet_manager.cc:1397] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20250901 20:02:25.168032 23307 raft_consensus.cc:357] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.168145 23307 raft_consensus.cc:383] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:25.168190 23307 raft_consensus.cc:738] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2634c0c7fa74861af82abdc0a076d97, State: Initialized, Role: FOLLOWER
I20250901 20:02:25.168322 23307 consensus_queue.cc:260] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.168599 23307 ts_tablet_manager.cc:1428] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:02:25.168560 23326 tablet_replica.cc:440] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } }
I20250901 20:02:25.168752 23326 tablet_replica.cc:443] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:02:25.266485 23326 raft_consensus.cc:491] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:02:25.266644 23326 raft_consensus.cc:513] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.267113 23326 leader_election.cc:290] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb295f74a05d4d3fa67899151432b65b (127.14.67.65:33435), a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941)
I20250901 20:02:25.267472 23090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0a88d6e9aa2460ea790ad6938fd3ba5" candidate_uuid: "c2634c0c7fa74861af82abdc0a076d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ace1800c014ff18b217adfb028f7a4" is_pre_election: true
I20250901 20:02:25.267535 22948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0a88d6e9aa2460ea790ad6938fd3ba5" candidate_uuid: "c2634c0c7fa74861af82abdc0a076d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb295f74a05d4d3fa67899151432b65b" is_pre_election: true
I20250901 20:02:25.267602 23090 raft_consensus.cc:2466] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2634c0c7fa74861af82abdc0a076d97 in term 0.
I20250901 20:02:25.267679 22948 raft_consensus.cc:2466] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2634c0c7fa74861af82abdc0a076d97 in term 0.
I20250901 20:02:25.267896 23163 leader_election.cc:304] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ace1800c014ff18b217adfb028f7a4, c2634c0c7fa74861af82abdc0a076d97; no voters:
I20250901 20:02:25.268061 23326 raft_consensus.cc:2802] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:02:25.268107 23326 raft_consensus.cc:491] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:02:25.268141 23326 raft_consensus.cc:3058] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:25.269260 23326 raft_consensus.cc:513] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.269657 23326 leader_election.cc:290] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [CANDIDATE]: Term 1 election: Requested vote from peers cb295f74a05d4d3fa67899151432b65b (127.14.67.65:33435), a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941)
I20250901 20:02:25.269927 23090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0a88d6e9aa2460ea790ad6938fd3ba5" candidate_uuid: "c2634c0c7fa74861af82abdc0a076d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ace1800c014ff18b217adfb028f7a4"
I20250901 20:02:25.269914 22948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0a88d6e9aa2460ea790ad6938fd3ba5" candidate_uuid: "c2634c0c7fa74861af82abdc0a076d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb295f74a05d4d3fa67899151432b65b"
I20250901 20:02:25.270025 22948 raft_consensus.cc:3058] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:25.270025 23090 raft_consensus.cc:3058] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:25.271032 23090 raft_consensus.cc:2466] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2634c0c7fa74861af82abdc0a076d97 in term 1.
I20250901 20:02:25.271061 22948 raft_consensus.cc:2466] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2634c0c7fa74861af82abdc0a076d97 in term 1.
I20250901 20:02:25.271346 23164 leader_election.cc:304] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2634c0c7fa74861af82abdc0a076d97, cb295f74a05d4d3fa67899151432b65b; no voters:
I20250901 20:02:25.271569 23326 raft_consensus.cc:2802] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:25.271682 23326 raft_consensus.cc:695] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [term 1 LEADER]: Becoming Leader. State: Replica: c2634c0c7fa74861af82abdc0a076d97, State: Running, Role: LEADER
I20250901 20:02:25.271837 23326 consensus_queue.cc:237] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.272554 23312 tablet_replica.cc:440] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: TxnStatusTablet state changed. Reason: New leader c2634c0c7fa74861af82abdc0a076d97. Latest consensus state: current_term: 1 leader_uuid: "c2634c0c7fa74861af82abdc0a076d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } }
I20250901 20:02:25.272648 23312 tablet_replica.cc:443] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: This TxnStatusTablet replica's current role is: LEADER
I20250901 20:02:25.273030 23349 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250901 20:02:25.273171 23349 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250901 20:02:25.273476 22788 catalog_manager.cc:5582] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2634c0c7fa74861af82abdc0a076d97 (127.14.67.67). New cstate: current_term: 1 leader_uuid: "c2634c0c7fa74861af82abdc0a076d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } health_report { overall_health: HEALTHY } } }
I20250901 20:02:25.293164 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.290543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:34857 (local address 127.0.0.1:32960)
0901 20:02:25.290762 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:25.290777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:25.290893 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:25.291136 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:25.291139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:25.291146 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0901 20:02:25.291342 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:25.291348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.292027 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.292030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:25.292425 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:25.292432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.292537 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.292581 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0901 20:02:25.292893 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0901 20:02:25.292908 (+ 15us) client_negotiation.cc:769] Sending connection context
0901 20:02:25.292968 (+ 60us) client_negotiation.cc:240] Negotiation successful
0901 20:02:25.293039 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20250901 20:02:25.293164 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.290634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.14.67.67:34857)
0901 20:02:25.290751 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:25.290755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:25.290830 (+ 75us) server_negotiation.cc:407] Connection header received
0901 20:02:25.291004 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:25.291007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:25.291049 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:25.291115 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:02:25.291439 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.291915 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.292528 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.292708 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.292747 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:02:25.292807 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:02:25.292873 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:25.292997 (+ 124us) server_negotiation.cc:299] Negotiation successful
0901 20:02:25.293038 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20250901 20:02:25.295317 22948 raft_consensus.cc:1273] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Refusing update from remote peer c2634c0c7fa74861af82abdc0a076d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:25.295336 23090 raft_consensus.cc:1273] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Refusing update from remote peer c2634c0c7fa74861af82abdc0a076d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:25.295719 23312 consensus_queue.cc:1035] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:25.295851 23326 consensus_queue.cc:1035] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:25.296994 23313 tablet_replica.cc:440] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4: TxnStatusTablet state changed. Reason: New leader c2634c0c7fa74861af82abdc0a076d97. Latest consensus state: current_term: 1 leader_uuid: "c2634c0c7fa74861af82abdc0a076d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } }
I20250901 20:02:25.297088 23313 tablet_replica.cc:443] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:02:25.297068 23314 tablet_replica.cc:440] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b: TxnStatusTablet state changed. Reason: New leader c2634c0c7fa74861af82abdc0a076d97. Latest consensus state: current_term: 1 leader_uuid: "c2634c0c7fa74861af82abdc0a076d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } }
I20250901 20:02:25.297158 23314 tablet_replica.cc:443] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:02:25.298291 23316 tablet_replica.cc:440] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2634c0c7fa74861af82abdc0a076d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } }
I20250901 20:02:25.298381 23316 tablet_replica.cc:443] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: This TxnStatusTablet replica's current role is: LEADER
I20250901 20:02:25.298514 23319 mvcc.cc:204] Tried to move back new op lower bound from 7195676447926300672 to 7195676447834963968. Current Snapshot: MvccSnapshot[applied={T|T < 7195676447926300672}]
I20250901 20:02:25.299686 23316 tablet_replica.cc:440] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2634c0c7fa74861af82abdc0a076d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } }
I20250901 20:02:25.299770 23316 tablet_replica.cc:443] T f0a88d6e9aa2460ea790ad6938fd3ba5 P c2634c0c7fa74861af82abdc0a076d97: This TxnStatusTablet replica's current role is: LEADER
I20250901 20:02:25.300262 23320 mvcc.cc:204] Tried to move back new op lower bound from 7195676447926300672 to 7195676447834963968. Current Snapshot: MvccSnapshot[applied={T|T < 7195676447926300672}]
I20250901 20:02:25.300417 23314 tablet_replica.cc:440] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2634c0c7fa74861af82abdc0a076d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } }
I20250901 20:02:25.300496 23314 tablet_replica.cc:443] T f0a88d6e9aa2460ea790ad6938fd3ba5 P cb295f74a05d4d3fa67899151432b65b: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:02:25.301019 23313 tablet_replica.cc:440] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2634c0c7fa74861af82abdc0a076d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } } }
I20250901 20:02:25.301106 23313 tablet_replica.cc:443] T f0a88d6e9aa2460ea790ad6938fd3ba5 P a9ace1800c014ff18b217adfb028f7a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20250901 20:02:25.307137 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.303279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.14.67.65:33435)
0901 20:02:25.303385 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:25.303389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:25.303491 (+ 102us) server_negotiation.cc:407] Connection header received
0901 20:02:25.303654 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:25.303658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:25.303715 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:25.303805 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:02:25.304288 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.304745 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.305656 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.305864 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.305941 (+ 77us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:02:25.306580 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:02:25.306725 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:25.306858 (+ 133us) server_negotiation.cc:299] Negotiation successful
0901 20:02:25.306928 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20250901 20:02:25.317391 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.314692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:34857 (local address 127.14.67.65:47413)
0901 20:02:25.314967 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:25.314980 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:25.315088 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:25.315330 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:25.315333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:25.315340 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0901 20:02:25.315566 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:25.315572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.316212 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.316218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:25.316657 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:25.316663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.316766 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.316811 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0901 20:02:25.317117 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0901 20:02:25.317125 (+ 8us) client_negotiation.cc:769] Sending connection context
0901 20:02:25.317187 (+ 62us) client_negotiation.cc:240] Negotiation successful
0901 20:02:25.317249 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":101,"threads_started":1}
I20250901 20:02:25.317373 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.314751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:47413 (local address 127.14.67.67:34857)
0901 20:02:25.314877 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:25.314883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:25.315052 (+ 169us) server_negotiation.cc:407] Connection header received
0901 20:02:25.315198 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:25.315208 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:25.315250 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:25.315313 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:02:25.315665 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.316111 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.316759 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.316935 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.316976 (+ 41us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:02:25.317033 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:02:25.317098 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:25.317209 (+ 111us) server_negotiation.cc:299] Negotiation successful
0901 20:02:25.317248 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250901 20:02:25.323999 23090 raft_consensus.cc:1273] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Refusing update from remote peer cb295f74a05d4d3fa67899151432b65b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:25.324131 23219 raft_consensus.cc:1273] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Refusing update from remote peer cb295f74a05d4d3fa67899151432b65b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:25.324429 23314 consensus_queue.cc:1035] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:25.324810 23337 consensus_queue.cc:1035] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:25.343721 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.340837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:44867 (local address 127.14.67.65:33435)
0901 20:02:25.341002 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:25.341006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:25.341117 (+ 111us) server_negotiation.cc:407] Connection header received
0901 20:02:25.341231 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:25.341233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:25.341272 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:25.341335 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:02:25.341754 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.342333 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.343226 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.343351 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.343386 (+ 35us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:02:25.343444 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:02:25.343504 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:25.343598 (+ 94us) server_negotiation.cc:299] Negotiation successful
0901 20:02:25.343632 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20250901 20:02:25.343779 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.340738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:33435 (local address 127.14.67.67:44867)
0901 20:02:25.341035 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:25.341050 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:25.341150 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:25.341354 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:25.341357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:25.341365 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0901 20:02:25.341645 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:25.341652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.342438 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.342441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:25.343119 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:25.343128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.343238 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.343278 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0901 20:02:25.343523 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0901 20:02:25.343527 (+ 4us) client_negotiation.cc:769] Sending connection context
0901 20:02:25.343589 (+ 62us) client_negotiation.cc:240] Negotiation successful
0901 20:02:25.343648 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
I20250901 20:02:25.372244 22787 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38326:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:25.372478 22787 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38326:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:25.375056 22787 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558: Sending DeleteTablet for 3 replicas of tablet aa817b7a1f5542da8f151fc3760d6cd0
I20250901 20:02:25.375795 23070 tablet_service.cc:1515] Processing DeleteTablet for tablet aa817b7a1f5542da8f151fc3760d6cd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:25 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:43328
I20250901 20:02:25.375854 22928 tablet_service.cc:1515] Processing DeleteTablet for tablet aa817b7a1f5542da8f151fc3760d6cd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:25 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:33360
I20250901 20:02:25.376031 23330 tablet_replica.cc:331] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4: stopping tablet replica
I20250901 20:02:25.376040 23331 tablet_replica.cc:331] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b: stopping tablet replica
I20250901 20:02:25.376147 23331 raft_consensus.cc:2241] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:02:25.376164 23330 raft_consensus.cc:2241] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:25.376295 23330 raft_consensus.cc:2270] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:25.376423 23331 raft_consensus.cc:2270] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:25.377238 23331 ts_tablet_manager.cc:1905] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:25.377480 23330 ts_tablet_manager.cc:1905] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:25.380079 23330 ts_tablet_manager.cc:1918] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:02:25.380092 23331 ts_tablet_manager.cc:1918] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:02:25.380153 23330 log.cc:1199] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/aa817b7a1f5542da8f151fc3760d6cd0
I20250901 20:02:25.380156 23331 log.cc:1199] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/wal/wals/aa817b7a1f5542da8f151fc3760d6cd0
I20250901 20:02:25.380414 23331 ts_tablet_manager.cc:1939] T aa817b7a1f5542da8f151fc3760d6cd0 P cb295f74a05d4d3fa67899151432b65b: Deleting consensus metadata
I20250901 20:02:25.380424 23330 ts_tablet_manager.cc:1939] T aa817b7a1f5542da8f151fc3760d6cd0 P a9ace1800c014ff18b217adfb028f7a4: Deleting consensus metadata
I20250901 20:02:25.380622 23210 tablet_service.cc:1515] Processing DeleteTablet for tablet aa817b7a1f5542da8f151fc3760d6cd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:25 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:32940
I20250901 20:02:25.380852 23329 tablet_replica.cc:331] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97: stopping tablet replica
I20250901 20:02:25.380967 23329 raft_consensus.cc:2241] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:25.381096 23329 raft_consensus.cc:2270] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:25.381641 22773 catalog_manager.cc:4928] TS a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941): tablet aa817b7a1f5542da8f151fc3760d6cd0 (table test-table [id=83d35f38cf0247ecb44908b92caabe9e]) successfully deleted
I20250901 20:02:25.381914 23329 ts_tablet_manager.cc:1905] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:25.384778 23329 ts_tablet_manager.cc:1918] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:02:25.384850 23329 log.cc:1199] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/aa817b7a1f5542da8f151fc3760d6cd0
I20250901 20:02:25.385106 23329 ts_tablet_manager.cc:1939] T aa817b7a1f5542da8f151fc3760d6cd0 P c2634c0c7fa74861af82abdc0a076d97: Deleting consensus metadata
I20250901 20:02:25.385744 22776 catalog_manager.cc:4928] TS c2634c0c7fa74861af82abdc0a076d97 (127.14.67.67:34857): tablet aa817b7a1f5542da8f151fc3760d6cd0 (table test-table [id=83d35f38cf0247ecb44908b92caabe9e]) successfully deleted
I20250901 20:02:25.385941 22774 catalog_manager.cc:4928] TS cb295f74a05d4d3fa67899151432b65b (127.14.67.65:33435): tablet aa817b7a1f5542da8f151fc3760d6cd0 (table test-table [id=83d35f38cf0247ecb44908b92caabe9e]) successfully deleted
Sep 01 20:02:25 dist-test-slave-2s4d krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 01 20:02:25 dist-test-slave-2s4d krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:25 dist-test-slave-2s4d krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
W20250901 20:02:25.411392 22838 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:25.403854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.14.67.126:34321)
0901 20:02:25.403999 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:25.404004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:25.404330 (+ 326us) server_negotiation.cc:407] Connection header received
0901 20:02:25.404472 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:25.404476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:25.404560 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:25.404664 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:25.405755 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.406632 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.407600 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.407782 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.411243 (+ 3461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59344: BlockingRecv error: recv got EOF from 127.0.0.1:59344 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 01 20:02:25 dist-test-slave-2s4d krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM
I20250901 20:02:25.422374 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.412912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.14.67.126:34321)
0901 20:02:25.413055 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:25.413060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:25.413483 (+ 423us) server_negotiation.cc:407] Connection header received
0901 20:02:25.413670 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:25.413674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:25.413727 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:25.413955 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:25.415041 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.415844 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.416618 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.416816 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.419207 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:25.419226 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:25.419229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:25.419256 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:25.420903 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:25.421350 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:25.421354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:25.421356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:25.421400 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:25.421607 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:25.421610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:25.421612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:25.421774 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:25.421896 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:25.422100 (+ 204us) server_negotiation.cc:299] Negotiation successful
0901 20:02:25.422207 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20250901 20:02:25.425227 22787 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 20:02:25.425693 22787 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:02:25.430703 22928 tablet_service.cc:1468] Processing CreateTablet for tablet 91ce235409ad4a8d81a30fdceb06430e (DEFAULT_TABLE table=test-table [id=278347d0ebeb4e46a009aa83b6bac78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:25.430819 23210 tablet_service.cc:1468] Processing CreateTablet for tablet 91ce235409ad4a8d81a30fdceb06430e (DEFAULT_TABLE table=test-table [id=278347d0ebeb4e46a009aa83b6bac78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:25.431001 22928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91ce235409ad4a8d81a30fdceb06430e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:25.431102 23210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91ce235409ad4a8d81a30fdceb06430e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:25.433431 23307 tablet_bootstrap.cc:492] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97: Bootstrap starting.
I20250901 20:02:25.433480 23308 tablet_bootstrap.cc:492] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b: Bootstrap starting.
I20250901 20:02:25.434378 23308 tablet_bootstrap.cc:654] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:25.434378 23307 tablet_bootstrap.cc:654] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:25.434787 23070 tablet_service.cc:1468] Processing CreateTablet for tablet 91ce235409ad4a8d81a30fdceb06430e (DEFAULT_TABLE table=test-table [id=278347d0ebeb4e46a009aa83b6bac78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:25.435060 23070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91ce235409ad4a8d81a30fdceb06430e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:25.435669 23307 tablet_bootstrap.cc:492] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97: No bootstrap required, opened a new log
I20250901 20:02:25.435751 23307 ts_tablet_manager.cc:1397] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250901 20:02:25.436208 23307 raft_consensus.cc:357] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.436326 23307 raft_consensus.cc:383] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:25.436365 23307 raft_consensus.cc:738] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2634c0c7fa74861af82abdc0a076d97, State: Initialized, Role: FOLLOWER
I20250901 20:02:25.436488 23307 consensus_queue.cc:260] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.436733 23307 ts_tablet_manager.cc:1428] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:02:25.437203 23306 tablet_bootstrap.cc:492] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4: Bootstrap starting.
I20250901 20:02:25.438198 23306 tablet_bootstrap.cc:654] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:25.440073 23308 tablet_bootstrap.cc:492] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b: No bootstrap required, opened a new log
I20250901 20:02:25.440164 23308 ts_tablet_manager.cc:1397] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250901 20:02:25.440608 23308 raft_consensus.cc:357] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.440707 23308 raft_consensus.cc:383] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:25.440740 23308 raft_consensus.cc:738] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb295f74a05d4d3fa67899151432b65b, State: Initialized, Role: FOLLOWER
I20250901 20:02:25.440857 23308 consensus_queue.cc:260] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.441114 23308 ts_tablet_manager.cc:1428] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250901 20:02:25.441572 23306 tablet_bootstrap.cc:492] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4: No bootstrap required, opened a new log
I20250901 20:02:25.441653 23306 ts_tablet_manager.cc:1397] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250901 20:02:25.442077 23306 raft_consensus.cc:357] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.442167 23306 raft_consensus.cc:383] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:25.442200 23306 raft_consensus.cc:738] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ace1800c014ff18b217adfb028f7a4, State: Initialized, Role: FOLLOWER
I20250901 20:02:25.442312 23306 consensus_queue.cc:260] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.444451 23306 ts_tablet_manager.cc:1428] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250901 20:02:25.566428 23326 raft_consensus.cc:491] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:02:25.566669 23326 raft_consensus.cc:513] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.567253 23326 leader_election.cc:290] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb295f74a05d4d3fa67899151432b65b (127.14.67.65:33435), a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941)
I20250901 20:02:25.567792 23090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91ce235409ad4a8d81a30fdceb06430e" candidate_uuid: "c2634c0c7fa74861af82abdc0a076d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ace1800c014ff18b217adfb028f7a4" is_pre_election: true
I20250901 20:02:25.567798 22948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91ce235409ad4a8d81a30fdceb06430e" candidate_uuid: "c2634c0c7fa74861af82abdc0a076d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb295f74a05d4d3fa67899151432b65b" is_pre_election: true
I20250901 20:02:25.567965 22948 raft_consensus.cc:2466] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2634c0c7fa74861af82abdc0a076d97 in term 0.
I20250901 20:02:25.567965 23090 raft_consensus.cc:2466] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2634c0c7fa74861af82abdc0a076d97 in term 0.
I20250901 20:02:25.568369 23163 leader_election.cc:304] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ace1800c014ff18b217adfb028f7a4, c2634c0c7fa74861af82abdc0a076d97; no voters:
I20250901 20:02:25.568616 23326 raft_consensus.cc:2802] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:02:25.568712 23326 raft_consensus.cc:491] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:02:25.568740 23326 raft_consensus.cc:3058] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:25.569730 23326 raft_consensus.cc:513] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.570163 23326 leader_election.cc:290] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [CANDIDATE]: Term 1 election: Requested vote from peers cb295f74a05d4d3fa67899151432b65b (127.14.67.65:33435), a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941)
I20250901 20:02:25.570547 22948 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91ce235409ad4a8d81a30fdceb06430e" candidate_uuid: "c2634c0c7fa74861af82abdc0a076d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb295f74a05d4d3fa67899151432b65b"
I20250901 20:02:25.570605 23090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91ce235409ad4a8d81a30fdceb06430e" candidate_uuid: "c2634c0c7fa74861af82abdc0a076d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ace1800c014ff18b217adfb028f7a4"
I20250901 20:02:25.570700 22948 raft_consensus.cc:3058] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:25.570719 23090 raft_consensus.cc:3058] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:25.571686 23090 raft_consensus.cc:2466] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2634c0c7fa74861af82abdc0a076d97 in term 1.
I20250901 20:02:25.571686 22948 raft_consensus.cc:2466] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2634c0c7fa74861af82abdc0a076d97 in term 1.
I20250901 20:02:25.572041 23163 leader_election.cc:304] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ace1800c014ff18b217adfb028f7a4, c2634c0c7fa74861af82abdc0a076d97; no voters:
I20250901 20:02:25.572275 23326 raft_consensus.cc:2802] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:25.572381 23326 raft_consensus.cc:695] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 1 LEADER]: Becoming Leader. State: Replica: c2634c0c7fa74861af82abdc0a076d97, State: Running, Role: LEADER
I20250901 20:02:25.572528 23326 consensus_queue.cc:237] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.574069 22787 catalog_manager.cc:5582] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2634c0c7fa74861af82abdc0a076d97 (127.14.67.67). New cstate: current_term: 1 leader_uuid: "c2634c0c7fa74861af82abdc0a076d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } health_report { overall_health: HEALTHY } } }
I20250901 20:02:25.596652 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.594192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.14.67.67:34857)
0901 20:02:25.594330 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:25.594334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:25.594348 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:02:25.594457 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:25.594459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:25.594527 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:25.594609 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:02:25.594926 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.595389 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.596013 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.596176 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.596217 (+ 41us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:02:25.596274 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:02:25.596348 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:25.596480 (+ 132us) server_negotiation.cc:299] Negotiation successful
0901 20:02:25.596525 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20250901 20:02:25.598737 23090 raft_consensus.cc:1273] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Refusing update from remote peer c2634c0c7fa74861af82abdc0a076d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:25.598764 22948 raft_consensus.cc:1273] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Refusing update from remote peer c2634c0c7fa74861af82abdc0a076d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:25.599136 23316 consensus_queue.cc:1035] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:25.599256 23326 consensus_queue.cc:1035] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:25.606391 22787 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59346:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:25.606602 22787 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59346:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:25.608860 22787 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558: Sending DeleteTablet for 3 replicas of tablet 91ce235409ad4a8d81a30fdceb06430e
I20250901 20:02:25.609476 23210 tablet_service.cc:1515] Processing DeleteTablet for tablet 91ce235409ad4a8d81a30fdceb06430e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:25 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:32940
I20250901 20:02:25.609529 23070 tablet_service.cc:1515] Processing DeleteTablet for tablet 91ce235409ad4a8d81a30fdceb06430e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:25 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:43328
I20250901 20:02:25.609546 22928 tablet_service.cc:1515] Processing DeleteTablet for tablet 91ce235409ad4a8d81a30fdceb06430e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:25 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:33360
I20250901 20:02:25.609647 23329 tablet_replica.cc:331] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97: stopping tablet replica
I20250901 20:02:25.609767 23329 raft_consensus.cc:2241] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:02:25.609844 23330 tablet_replica.cc:331] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4: stopping tablet replica
I20250901 20:02:25.609930 23329 raft_consensus.cc:2270] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:25.609978 23330 raft_consensus.cc:2241] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:25.610119 23330 raft_consensus.cc:2270] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:25.610316 23331 tablet_replica.cc:331] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b: stopping tablet replica
I20250901 20:02:25.610441 23331 raft_consensus.cc:2241] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:25.610598 23331 raft_consensus.cc:2270] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:25.610702 22787 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 20:02:25.611002 23329 ts_tablet_manager.cc:1905] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 20:02:25.611107 22787 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:02:25.611440 23330 ts_tablet_manager.cc:1905] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:25.613080 23329 ts_tablet_manager.cc:1918] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:25.613148 23329 log.cc:1199] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/91ce235409ad4a8d81a30fdceb06430e
I20250901 20:02:25.613404 23329 ts_tablet_manager.cc:1939] T 91ce235409ad4a8d81a30fdceb06430e P c2634c0c7fa74861af82abdc0a076d97: Deleting consensus metadata
I20250901 20:02:25.614054 22776 catalog_manager.cc:4928] TS c2634c0c7fa74861af82abdc0a076d97 (127.14.67.67:34857): tablet 91ce235409ad4a8d81a30fdceb06430e (table test-table [id=278347d0ebeb4e46a009aa83b6bac78b]) successfully deleted
I20250901 20:02:25.614607 23331 ts_tablet_manager.cc:1905] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:25.616146 23210 tablet_service.cc:1468] Processing CreateTablet for tablet 26031ebd269545fab4edc16ccdfa2bca (DEFAULT_TABLE table=test-table [id=76010dbaffa24249bfed3e1a917f156a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:25.616367 23210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26031ebd269545fab4edc16ccdfa2bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:25.616566 23331 ts_tablet_manager.cc:1918] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:25.616633 23331 log.cc:1199] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-0/wal/wals/91ce235409ad4a8d81a30fdceb06430e
I20250901 20:02:25.616849 23331 ts_tablet_manager.cc:1939] T 91ce235409ad4a8d81a30fdceb06430e P cb295f74a05d4d3fa67899151432b65b: Deleting consensus metadata
I20250901 20:02:25.617422 22774 catalog_manager.cc:4928] TS cb295f74a05d4d3fa67899151432b65b (127.14.67.65:33435): tablet 91ce235409ad4a8d81a30fdceb06430e (table test-table [id=278347d0ebeb4e46a009aa83b6bac78b]) successfully deleted
I20250901 20:02:25.617619 22928 tablet_service.cc:1468] Processing CreateTablet for tablet 26031ebd269545fab4edc16ccdfa2bca (DEFAULT_TABLE table=test-table [id=76010dbaffa24249bfed3e1a917f156a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:25.617759 23070 tablet_service.cc:1468] Processing CreateTablet for tablet 26031ebd269545fab4edc16ccdfa2bca (DEFAULT_TABLE table=test-table [id=76010dbaffa24249bfed3e1a917f156a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:25.617828 22928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26031ebd269545fab4edc16ccdfa2bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:25.618017 23070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26031ebd269545fab4edc16ccdfa2bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:25.618170 23307 tablet_bootstrap.cc:492] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97: Bootstrap starting.
I20250901 20:02:25.619014 23307 tablet_bootstrap.cc:654] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:25.619755 23308 tablet_bootstrap.cc:492] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b: Bootstrap starting.
I20250901 20:02:25.619982 23306 tablet_bootstrap.cc:492] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4: Bootstrap starting.
I20250901 20:02:25.620544 23308 tablet_bootstrap.cc:654] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:25.620855 23306 tablet_bootstrap.cc:654] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:25.621367 23307 tablet_bootstrap.cc:492] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97: No bootstrap required, opened a new log
I20250901 20:02:25.621451 23307 ts_tablet_manager.cc:1397] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250901 20:02:25.621472 23308 tablet_bootstrap.cc:492] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b: No bootstrap required, opened a new log
I20250901 20:02:25.621526 23308 ts_tablet_manager.cc:1397] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250901 20:02:25.621547 23330 ts_tablet_manager.cc:1918] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:25.621603 23330 log.cc:1199] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/91ce235409ad4a8d81a30fdceb06430e
I20250901 20:02:25.621836 23330 ts_tablet_manager.cc:1939] T 91ce235409ad4a8d81a30fdceb06430e P a9ace1800c014ff18b217adfb028f7a4: Deleting consensus metadata
I20250901 20:02:25.621901 23307 raft_consensus.cc:357] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.621919 23308 raft_consensus.cc:357] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.621999 23307 raft_consensus.cc:383] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:25.622017 23308 raft_consensus.cc:383] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:25.622032 23307 raft_consensus.cc:738] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2634c0c7fa74861af82abdc0a076d97, State: Initialized, Role: FOLLOWER
I20250901 20:02:25.622051 23308 raft_consensus.cc:738] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb295f74a05d4d3fa67899151432b65b, State: Initialized, Role: FOLLOWER
I20250901 20:02:25.622152 23307 consensus_queue.cc:260] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.622160 23308 consensus_queue.cc:260] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.622433 23307 ts_tablet_manager.cc:1428] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250901 20:02:25.622445 23308 ts_tablet_manager.cc:1428] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:02:25.622468 22773 catalog_manager.cc:4928] TS a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941): tablet 91ce235409ad4a8d81a30fdceb06430e (table test-table [id=278347d0ebeb4e46a009aa83b6bac78b]) successfully deleted
I20250901 20:02:25.623252 23306 tablet_bootstrap.cc:492] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4: No bootstrap required, opened a new log
I20250901 20:02:25.623333 23306 ts_tablet_manager.cc:1397] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250901 20:02:25.623680 23306 raft_consensus.cc:357] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.623764 23306 raft_consensus.cc:383] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:25.623800 23306 raft_consensus.cc:738] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ace1800c014ff18b217adfb028f7a4, State: Initialized, Role: FOLLOWER
I20250901 20:02:25.623929 23306 consensus_queue.cc:260] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.624194 23306 ts_tablet_manager.cc:1428] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250901 20:02:25.793195 23361 raft_consensus.cc:491] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:02:25.793390 23361 raft_consensus.cc:513] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.793998 23361 leader_election.cc:290] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941), c2634c0c7fa74861af82abdc0a076d97 (127.14.67.67:34857)
I20250901 20:02:25.794606 23090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26031ebd269545fab4edc16ccdfa2bca" candidate_uuid: "cb295f74a05d4d3fa67899151432b65b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ace1800c014ff18b217adfb028f7a4" is_pre_election: true
I20250901 20:02:25.794631 23219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26031ebd269545fab4edc16ccdfa2bca" candidate_uuid: "cb295f74a05d4d3fa67899151432b65b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2634c0c7fa74861af82abdc0a076d97" is_pre_election: true
I20250901 20:02:25.794802 23219 raft_consensus.cc:2466] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb295f74a05d4d3fa67899151432b65b in term 0.
I20250901 20:02:25.794802 23090 raft_consensus.cc:2466] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb295f74a05d4d3fa67899151432b65b in term 0.
I20250901 20:02:25.795149 22882 leader_election.cc:304] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2634c0c7fa74861af82abdc0a076d97, cb295f74a05d4d3fa67899151432b65b; no voters:
I20250901 20:02:25.795362 23361 raft_consensus.cc:2802] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:02:25.795428 23361 raft_consensus.cc:491] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:02:25.795470 23361 raft_consensus.cc:3058] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:25.796388 23361 raft_consensus.cc:513] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.796741 23361 leader_election.cc:290] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [CANDIDATE]: Term 1 election: Requested vote from peers a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941), c2634c0c7fa74861af82abdc0a076d97 (127.14.67.67:34857)
I20250901 20:02:25.797068 23090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26031ebd269545fab4edc16ccdfa2bca" candidate_uuid: "cb295f74a05d4d3fa67899151432b65b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ace1800c014ff18b217adfb028f7a4"
I20250901 20:02:25.797134 23219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26031ebd269545fab4edc16ccdfa2bca" candidate_uuid: "cb295f74a05d4d3fa67899151432b65b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2634c0c7fa74861af82abdc0a076d97"
I20250901 20:02:25.797159 23090 raft_consensus.cc:3058] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:25.797245 23219 raft_consensus.cc:3058] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:25.797936 23090 raft_consensus.cc:2466] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb295f74a05d4d3fa67899151432b65b in term 1.
I20250901 20:02:25.798072 23219 raft_consensus.cc:2466] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb295f74a05d4d3fa67899151432b65b in term 1.
I20250901 20:02:25.798213 22879 leader_election.cc:304] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ace1800c014ff18b217adfb028f7a4, cb295f74a05d4d3fa67899151432b65b; no voters:
I20250901 20:02:25.798377 23361 raft_consensus.cc:2802] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:25.798455 23361 raft_consensus.cc:695] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 1 LEADER]: Becoming Leader. State: Replica: cb295f74a05d4d3fa67899151432b65b, State: Running, Role: LEADER
I20250901 20:02:25.798610 23361 consensus_queue.cc:237] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } }
I20250901 20:02:25.800272 22789 catalog_manager.cc:5582] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b reported cstate change: term changed from 0 to 1, leader changed from <none> to cb295f74a05d4d3fa67899151432b65b (127.14.67.65). New cstate: current_term: 1 leader_uuid: "cb295f74a05d4d3fa67899151432b65b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb295f74a05d4d3fa67899151432b65b" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 33435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 } health_report { overall_health: UNKNOWN } } }
I20250901 20:02:25.818552 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.815840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.14.67.65:33435)
0901 20:02:25.815950 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:25.815953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:25.816072 (+ 119us) server_negotiation.cc:407] Connection header received
0901 20:02:25.816192 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:25.816194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:25.816234 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:25.816292 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:02:25.816600 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.817069 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:25.817768 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:25.818018 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:25.818075 (+ 57us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:02:25.818166 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:02:25.818243 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:25.818371 (+ 128us) server_negotiation.cc:299] Negotiation successful
0901 20:02:25.818420 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20250901 20:02:25.824138 23090 raft_consensus.cc:1273] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Refusing update from remote peer cb295f74a05d4d3fa67899151432b65b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:25.824138 23219 raft_consensus.cc:1273] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Refusing update from remote peer cb295f74a05d4d3fa67899151432b65b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:25.824553 23314 consensus_queue.cc:1035] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ace1800c014ff18b217adfb028f7a4" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:25.824688 23361 consensus_queue.cc:1035] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2634c0c7fa74861af82abdc0a076d97" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:25.856526 22789 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59346:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:25.856673 22789 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59346:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:25.858811 22789 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b802f41ef7be4bda914bb8397fd1c558: Sending DeleteTablet for 3 replicas of tablet 26031ebd269545fab4edc16ccdfa2bca
I20250901 20:02:25.859380 22928 tablet_service.cc:1515] Processing DeleteTablet for tablet 26031ebd269545fab4edc16ccdfa2bca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:25 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:33360
I20250901 20:02:25.859436 23070 tablet_service.cc:1515] Processing DeleteTablet for tablet 26031ebd269545fab4edc16ccdfa2bca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:25 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:43328
I20250901 20:02:25.859568 23331 tablet_replica.cc:331] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b: stopping tablet replica
I20250901 20:02:25.859611 23330 tablet_replica.cc:331] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4: stopping tablet replica
I20250901 20:02:25.859683 23331 raft_consensus.cc:2241] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:02:25.859694 23210 tablet_service.cc:1515] Processing DeleteTablet for tablet 26031ebd269545fab4edc16ccdfa2bca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:25 UTC) from {username='oryx', principal='oryx/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:32940
I20250901 20:02:25.859730 23330 raft_consensus.cc:2241] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:25.859851 23329 tablet_replica.cc:331] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97: stopping tablet replica
I20250901 20:02:25.859851 23330 raft_consensus.cc:2270] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:25.859880 23331 raft_consensus.cc:2270] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:25.859966 23329 raft_consensus.cc:2241] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:25.860073 23329 raft_consensus.cc:2270] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:25.860376 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 22863
I20250901 20:02:25.860705 23331 ts_tablet_manager.cc:1905] T 26031ebd269545fab4edc16ccdfa2bca P cb295f74a05d4d3fa67899151432b65b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:25.860913 23330 ts_tablet_manager.cc:1905] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:25.861790 23329 ts_tablet_manager.cc:1905] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:25.863664 23329 ts_tablet_manager.cc:1918] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:02:25.863734 23329 log.cc:1199] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/26031ebd269545fab4edc16ccdfa2bca
I20250901 20:02:25.863746 23330 ts_tablet_manager.cc:1918] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250901 20:02:25.863816 23330 log.cc:1199] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/26031ebd269545fab4edc16ccdfa2bca
I20250901 20:02:25.864004 23329 ts_tablet_manager.cc:1939] T 26031ebd269545fab4edc16ccdfa2bca P c2634c0c7fa74861af82abdc0a076d97: Deleting consensus metadata
I20250901 20:02:25.864055 23330 ts_tablet_manager.cc:1939] T 26031ebd269545fab4edc16ccdfa2bca P a9ace1800c014ff18b217adfb028f7a4: Deleting consensus metadata
I20250901 20:02:25.864615 22776 catalog_manager.cc:4928] TS c2634c0c7fa74861af82abdc0a076d97 (127.14.67.67:34857): tablet 26031ebd269545fab4edc16ccdfa2bca (table test-table [id=76010dbaffa24249bfed3e1a917f156a]) successfully deleted
I20250901 20:02:25.864902 22773 catalog_manager.cc:4928] TS a9ace1800c014ff18b217adfb028f7a4 (127.14.67.66:40941): tablet 26031ebd269545fab4edc16ccdfa2bca (table test-table [id=76010dbaffa24249bfed3e1a917f156a]) successfully deleted
W20250901 20:02:25.870019 22774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.67.65:33435 (error 108)
I20250901 20:02:25.870293 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 23005
I20250901 20:02:25.871985 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:25.871411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:33435 (local address 127.0.0.1:55554)
0901 20:02:25.871772 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:25.871848 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.65:33435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":73,"threads_started":1}
W20250901 20:02:25.872099 22774 catalog_manager.cc:4671] TS cb295f74a05d4d3fa67899151432b65b (127.14.67.65:33435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26031ebd269545fab4edc16ccdfa2bca: Network error: Client connection negotiation failed: client connection to 127.14.67.65:33435: connect: Connection refused (error 111)
I20250901 20:02:25.877264 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 23147
I20250901 20:02:25.884076 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 22757
2025-09-01T20:02:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4205 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:02:25 dist-test-slave-2s4d krb5kdc[23393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:02:25 dist-test-slave-2s4d krb5kdc[23393](info): set up 2 sockets
Sep 01 20:02:25 dist-test-slave-2s4d krb5kdc[23393](info): commencing operation
krb5kdc: starting...
W20250901 20:02:27.876128 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.969s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:02:27 dist-test-slave-2s4d krb5kdc[23393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:02:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:02:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:28.011255 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:35687
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:35687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:28.112232 23409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:28.112442 23409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:28.112524 23409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:28.115542 23409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:28.115598 23409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:28.115644 23409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:28.115687 23409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:28.115728 23409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:02:28.119714 23409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:35687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:28.120625 23409 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:28.121385 23409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:28.126638 23415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:28.126652 23417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:28.126740 23409 server_base.cc:1047] running on GCE node
W20250901 20:02:28.126652 23414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:28.127547 23409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:28.128634 23409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:28.129848 23409 hybrid_clock.cc:648] HybridClock initialized: now 1756756948129833 us; error 31 us; skew 500 ppm
W20250901 20:02:28.133258 23409 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-09-01T20:02:28Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2253 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:02:28 dist-test-slave-2s4d krb5kdc[23422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:02:28 dist-test-slave-2s4d krb5kdc[23422](info): set up 2 sockets
Sep 01 20:02:28 dist-test-slave-2s4d krb5kdc[23422](info): commencing operation
krb5kdc: starting...
W20250901 20:02:30.151997 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.992s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:02:30 dist-test-slave-2s4d krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:02:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:30.292157 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:45747
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:45747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:30.397899 23438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:30.398105 23438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:30.398157 23438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:30.401214 23438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:30.401268 23438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:30.401291 23438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:30.401309 23438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:30.401332 23438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:02:30.405297 23438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:45747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:45747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:30.406215 23438 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:30.407003 23438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:30.412232 23444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:30.412249 23443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:30.412343 23438 server_base.cc:1047] running on GCE node
W20250901 20:02:30.412490 23446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:30.412931 23438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:30.413785 23438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:30.414947 23438 hybrid_clock.cc:648] HybridClock initialized: now 1756756950414935 us; error 27 us; skew 500 ppm
Sep 01 20:02:30 dist-test-slave-2s4d krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756950, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:30.417527 23438 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:02:30.418469 23438 webserver.cc:480] Webserver started at http://127.14.67.126:41391/ using document root <none> and password file <none>
I20250901 20:02:30.419019 23438 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:30.419072 23438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:30.419265 23438 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:30.420897 23438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "c08471d881a44929a3d21048a479af09"
format_stamp: "Formatted at 2025-09-01 20:02:30 on dist-test-slave-2s4d"
server_key: "22715ab0e9446d27eac83250760509ba"
server_key_iv: "3b84a0c65837577904db66045e1ca814"
server_key_version: "encryptionkey@0"
I20250901 20:02:30.421365 23438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "c08471d881a44929a3d21048a479af09"
format_stamp: "Formatted at 2025-09-01 20:02:30 on dist-test-slave-2s4d"
server_key: "22715ab0e9446d27eac83250760509ba"
server_key_iv: "3b84a0c65837577904db66045e1ca814"
server_key_version: "encryptionkey@0"
I20250901 20:02:30.424801 23438 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20250901 20:02:30.426990 23453 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:30.428258 23438 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250901 20:02:30.428403 23438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "c08471d881a44929a3d21048a479af09"
format_stamp: "Formatted at 2025-09-01 20:02:30 on dist-test-slave-2s4d"
server_key: "22715ab0e9446d27eac83250760509ba"
server_key_iv: "3b84a0c65837577904db66045e1ca814"
server_key_version: "encryptionkey@0"
I20250901 20:02:30.428529 23438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:30.443360 23438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:30.561693 23438 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:30.561900 23438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:30.570246 23438 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:45747
I20250901 20:02:30.570246 23505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:45747 every 8 connection(s)
I20250901 20:02:30.571321 23438 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:02:30.573684 23506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:30.578810 23506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09: Bootstrap starting.
I20250901 20:02:30.580605 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 23438
I20250901 20:02:30.580752 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:02:30.581010 14605 external_mini_cluster.cc:1468] Setting key 085b709ac36e470dc0e2187a5c2f2390
I20250901 20:02:30.581140 23506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:30.581856 23506 log.cc:826] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:30.583644 23506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09: No bootstrap required, opened a new log
I20250901 20:02:30.585809 23506 raft_consensus.cc:357] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08471d881a44929a3d21048a479af09" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 45747 } }
I20250901 20:02:30.585986 23506 raft_consensus.cc:383] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:30.586071 23506 raft_consensus.cc:738] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08471d881a44929a3d21048a479af09, State: Initialized, Role: FOLLOWER
I20250901 20:02:30.586571 23506 consensus_queue.cc:260] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08471d881a44929a3d21048a479af09" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 45747 } }
I20250901 20:02:30.586702 23506 raft_consensus.cc:397] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:02:30.586771 23506 raft_consensus.cc:491] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:02:30.586871 23506 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:30.587636 23506 raft_consensus.cc:513] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08471d881a44929a3d21048a479af09" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 45747 } }
Sep 01 20:02:30 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:02:30.587963 23506 leader_election.cc:304] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c08471d881a44929a3d21048a479af09; no voters:
I20250901 20:02:30.588289 23506 leader_election.cc:290] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:02:30.588392 23511 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:30.588601 23511 raft_consensus.cc:695] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 1 LEADER]: Becoming Leader. State: Replica: c08471d881a44929a3d21048a479af09, State: Running, Role: LEADER
I20250901 20:02:30.588930 23511 consensus_queue.cc:237] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08471d881a44929a3d21048a479af09" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 45747 } }
I20250901 20:02:30.589504 23506 sys_catalog.cc:564] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:30.590152 23511 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c08471d881a44929a3d21048a479af09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08471d881a44929a3d21048a479af09" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 45747 } } }
I20250901 20:02:30.590363 23511 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:30.590551 23512 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c08471d881a44929a3d21048a479af09. Latest consensus state: current_term: 1 leader_uuid: "c08471d881a44929a3d21048a479af09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08471d881a44929a3d21048a479af09" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 45747 } } }
I20250901 20:02:30.590701 23512 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:30.590803 23520 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:02:30.594563 23520 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:02:30.595166 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:30.582044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.14.67.126:45747)
0901 20:02:30.582465 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:30.582478 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:30.582543 (+ 65us) server_negotiation.cc:407] Connection header received
0901 20:02:30.583079 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:30.583093 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:30.583352 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:30.583629 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:30.584510 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:30.585246 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:30.585844 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:30.586112 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:30.588408 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:30.588429 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:30.588443 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:30.588485 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:30.592471 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:30.592865 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:30.592871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:30.592876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:30.592959 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:30.593209 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:30.593214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:30.593217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:30.593592 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:30.593771 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:30.594105 (+ 334us) server_negotiation.cc:299] Negotiation successful
0901 20:02:30.594351 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"spinlock_wait_cycles":43392,"thread_start_us":161,"threads_started":1}
I20250901 20:02:30.599802 23520 catalog_manager.cc:1349] Generated new cluster ID: 579943c107a84c30927b9655715267f1
I20250901 20:02:30.599874 23520 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:02:30.609151 23520 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:02:30.610323 23520 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:02:30.623088 23520 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09: Generated new TSK 0
I20250901 20:02:30.623678 23520 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:30.679129 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:45747
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:30.813340 23534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:30.813527 23534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:30.813588 23534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:30.816641 23534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:30.816694 23534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:30.816803 23534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:02:30.820703 23534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:45747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:30.821622 23534 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:30.822399 23534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:30.828433 23539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:30.828475 23540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:30.828528 23542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:30.828820 23534 server_base.cc:1047] running on GCE node
I20250901 20:02:30.829327 23534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:30.830019 23534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:30.831199 23534 hybrid_clock.cc:648] HybridClock initialized: now 1756756950831157 us; error 64 us; skew 500 ppm
Sep 01 20:02:30 dist-test-slave-2s4d krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756950, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:30.834854 23534 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:02:30.835879 23534 webserver.cc:480] Webserver started at http://127.14.67.65:40947/ using document root <none> and password file <none>
I20250901 20:02:30.836416 23534 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:30.836469 23534 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:30.836644 23534 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:30.838320 23534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "6116f165ceda4efabcf6dfc90fe683d1"
format_stamp: "Formatted at 2025-09-01 20:02:30 on dist-test-slave-2s4d"
server_key: "69cc6d34d3c09e7f3ababd698ee4e396"
server_key_iv: "56b053fd1da3c66dfe653b6b6d627583"
server_key_version: "encryptionkey@0"
I20250901 20:02:30.838793 23534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "6116f165ceda4efabcf6dfc90fe683d1"
format_stamp: "Formatted at 2025-09-01 20:02:30 on dist-test-slave-2s4d"
server_key: "69cc6d34d3c09e7f3ababd698ee4e396"
server_key_iv: "56b053fd1da3c66dfe653b6b6d627583"
server_key_version: "encryptionkey@0"
I20250901 20:02:30.842021 23534 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250901 20:02:30.844424 23549 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:30.845886 23534 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250901 20:02:30.846050 23534 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "6116f165ceda4efabcf6dfc90fe683d1"
format_stamp: "Formatted at 2025-09-01 20:02:30 on dist-test-slave-2s4d"
server_key: "69cc6d34d3c09e7f3ababd698ee4e396"
server_key_iv: "56b053fd1da3c66dfe653b6b6d627583"
server_key_version: "encryptionkey@0"
I20250901 20:02:30.846192 23534 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:30.861681 23534 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:30.865265 23534 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:30.865494 23534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:30.866174 23534 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:30.867273 23534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:30.867329 23534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:30.867394 23534 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:30.867429 23534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:30.877668 23534 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:46539
I20250901 20:02:30.877691 23662 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:46539 every 8 connection(s)
I20250901 20:02:30.878654 23534 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:02:30.885663 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 23534
I20250901 20:02:30.885891 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:02:30.886253 14605 external_mini_cluster.cc:1468] Setting key 43e6471ef9eab45510909743a4cec9bc
Sep 01 20:02:30 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756950, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:02:30.892760 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:30.880482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:38921 (local address 127.14.67.126:45747)
0901 20:02:30.880639 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:30.880643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:30.881283 (+ 640us) server_negotiation.cc:407] Connection header received
0901 20:02:30.882164 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:30.882169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:30.882251 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:30.882373 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:30.883768 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:30.884302 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:30.885427 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:30.885635 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:30.888342 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:30.888359 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:30.888361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:30.888388 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:30.890122 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:30.891005 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:30.891011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:30.891016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:30.891073 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:30.891393 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:30.891398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:30.891403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:30.891649 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:30.891778 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:30.892286 (+ 508us) server_negotiation.cc:299] Negotiation successful
0901 20:02:30.892472 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20250901 20:02:30.893241 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:30.880702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:38921)
0901 20:02:30.881122 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:30.881168 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:30.881890 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:30.882570 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:30.882577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:30.883028 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:30.883586 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:30.883599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:30.884417 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:30.884419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:30.885275 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:30.885283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:30.885498 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:30.886162 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:30.886184 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:30.888147 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:30.890293 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:30.890299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:30.890315 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:30.890888 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:30.891172 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:30.891175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:30.891176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:30.891298 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:30.891773 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:30.891777 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:30.892003 (+ 226us) client_negotiation.cc:769] Sending connection context
0901 20:02:30.892172 (+ 169us) client_negotiation.cc:240] Negotiation successful
0901 20:02:30.892451 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":102,"threads_started":1}
I20250901 20:02:30.894856 23663 heartbeater.cc:344] Connected to a master server at 127.14.67.126:45747
I20250901 20:02:30.895228 23663 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:30.895912 23663 heartbeater.cc:507] Master 127.14.67.126:45747 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
I20250901 20:02:30.897677 23470 ts_manager.cc:194] Registered new tserver with Master: 6116f165ceda4efabcf6dfc90fe683d1 (127.14.67.65:46539)
I20250901 20:02:30.899127 23470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:38921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:30.937494 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:45747
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:31.036130 23670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:31.036296 23670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:31.036368 23670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:31.039364 23670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:31.039413 23670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:31.039522 23670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:02:31.043397 23670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:45747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:31.044275 23670 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:31.045017 23670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:31.051115 23675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:31.051146 23676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:31.051170 23678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:31.051432 23670 server_base.cc:1047] running on GCE node
I20250901 20:02:31.051795 23670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:31.052302 23670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:31.053462 23670 hybrid_clock.cc:648] HybridClock initialized: now 1756756951053452 us; error 33 us; skew 500 ppm
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756951, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:31.056313 23670 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:02:31.057219 23670 webserver.cc:480] Webserver started at http://127.14.67.66:44613/ using document root <none> and password file <none>
I20250901 20:02:31.057709 23670 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:31.057762 23670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:31.057929 23670 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:31.059535 23670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "8612a8b2186c4fb69ff302ae14e64e89"
format_stamp: "Formatted at 2025-09-01 20:02:31 on dist-test-slave-2s4d"
server_key: "cd025b6ac1a5ef495fe9353eacd30a79"
server_key_iv: "940fcf6065d088a807dab5de0e01cde5"
server_key_version: "encryptionkey@0"
I20250901 20:02:31.059975 23670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "8612a8b2186c4fb69ff302ae14e64e89"
format_stamp: "Formatted at 2025-09-01 20:02:31 on dist-test-slave-2s4d"
server_key: "cd025b6ac1a5ef495fe9353eacd30a79"
server_key_iv: "940fcf6065d088a807dab5de0e01cde5"
server_key_version: "encryptionkey@0"
I20250901 20:02:31.063311 23670 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20250901 20:02:31.065532 23685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:31.067116 23670 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250901 20:02:31.067248 23670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "8612a8b2186c4fb69ff302ae14e64e89"
format_stamp: "Formatted at 2025-09-01 20:02:31 on dist-test-slave-2s4d"
server_key: "cd025b6ac1a5ef495fe9353eacd30a79"
server_key_iv: "940fcf6065d088a807dab5de0e01cde5"
server_key_version: "encryptionkey@0"
I20250901 20:02:31.067370 23670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:31.085539 23670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:31.088770 23670 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:31.088980 23670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:31.089596 23670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:31.090701 23670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:31.090764 23670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:31.090824 23670 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:31.090857 23670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:31.102123 23670 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:44511
I20250901 20:02:31.102182 23798 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:44511 every 8 connection(s)
I20250901 20:02:31.103289 23670 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:02:31.112515 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 23670
I20250901 20:02:31.112635 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756951, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:02:31.112888 14605 external_mini_cluster.cc:1468] Setting key e7287140eb8fc56375c31f1486f92053
I20250901 20:02:31.117789 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:31.105138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:48171 (local address 127.14.67.126:45747)
0901 20:02:31.105307 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:31.105313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:31.106047 (+ 734us) server_negotiation.cc:407] Connection header received
0901 20:02:31.107105 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:31.107107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:31.107159 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:31.107248 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:31.108858 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.109489 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:31.110870 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.111034 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:31.113566 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:31.113581 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:31.113585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:31.113620 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:31.115589 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:31.116207 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:31.116209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:31.116211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:31.116249 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:31.116555 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:31.116557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:31.116558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:31.116696 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:31.116767 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:31.117445 (+ 678us) server_negotiation.cc:299] Negotiation successful
0901 20:02:31.117591 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20250901 20:02:31.118566 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:31.105313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:48171)
0901 20:02:31.105875 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:31.105920 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:31.106891 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:31.107466 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:31.107475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:31.108110 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:31.108657 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:31.108673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.109655 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:31.109659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:31.110739 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:31.110749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.110939 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:31.111627 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:31.111651 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:31.113426 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:31.115718 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:31.115723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:31.115730 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:31.116091 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:31.116334 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:31.116339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:31.116342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:31.116470 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:31.116789 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:31.116797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:31.117085 (+ 288us) client_negotiation.cc:769] Sending connection context
0901 20:02:31.117322 (+ 237us) client_negotiation.cc:240] Negotiation successful
0901 20:02:31.117634 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":127,"threads_started":1}
I20250901 20:02:31.119674 23799 heartbeater.cc:344] Connected to a master server at 127.14.67.126:45747
I20250901 20:02:31.119907 23799 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:31.120402 23799 heartbeater.cc:507] Master 127.14.67.126:45747 requested a full tablet report, sending...
I20250901 20:02:31.121378 23470 ts_manager.cc:194] Registered new tserver with Master: 8612a8b2186c4fb69ff302ae14e64e89 (127.14.67.66:44511)
I20250901 20:02:31.121924 23470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:48171
WARNING: no policy specified for kudu/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:31.164096 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:45747
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:31.266541 23806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:31.266742 23806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:31.266829 23806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:31.269855 23806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:31.269914 23806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:31.270033 23806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:02:31.274052 23806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:45747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:31.275017 23806 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:31.275753 23806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:31.281759 23811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:31.281806 23812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:31.281759 23814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:31.281881 23806 server_base.cc:1047] running on GCE node
I20250901 20:02:31.282315 23806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:31.282873 23806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:31.284047 23806 hybrid_clock.cc:648] HybridClock initialized: now 1756756951283978 us; error 77 us; skew 500 ppm
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756951, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:31.286806 23806 init.cc:377] Logged in from keytab as kudu/127.14.67.67@KRBTEST.COM (short username kudu)
I20250901 20:02:31.287777 23806 webserver.cc:480] Webserver started at http://127.14.67.67:33611/ using document root <none> and password file <none>
I20250901 20:02:31.288291 23806 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:31.288349 23806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:31.288523 23806 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:31.290138 23806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "9334d040d7e240d886ad5dd74e1a81df"
format_stamp: "Formatted at 2025-09-01 20:02:31 on dist-test-slave-2s4d"
server_key: "c857a2859869f9e13e6c5f7e4bc09803"
server_key_iv: "99b459a19cc8ee16a61371a012b75f95"
server_key_version: "encryptionkey@0"
I20250901 20:02:31.290644 23806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "9334d040d7e240d886ad5dd74e1a81df"
format_stamp: "Formatted at 2025-09-01 20:02:31 on dist-test-slave-2s4d"
server_key: "c857a2859869f9e13e6c5f7e4bc09803"
server_key_iv: "99b459a19cc8ee16a61371a012b75f95"
server_key_version: "encryptionkey@0"
I20250901 20:02:31.294257 23806 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250901 20:02:31.296373 23821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:31.297302 23806 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:31.297413 23806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "9334d040d7e240d886ad5dd74e1a81df"
format_stamp: "Formatted at 2025-09-01 20:02:31 on dist-test-slave-2s4d"
server_key: "c857a2859869f9e13e6c5f7e4bc09803"
server_key_iv: "99b459a19cc8ee16a61371a012b75f95"
server_key_version: "encryptionkey@0"
I20250901 20:02:31.297492 23806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:31.319895 23806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:31.322651 23806 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:31.322824 23806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:31.323343 23806 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:31.324177 23806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:31.324224 23806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:31.324282 23806 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:31.324313 23806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:31.334762 23806 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:41069
I20250901 20:02:31.334833 23934 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:41069 every 8 connection(s)
I20250901 20:02:31.335642 23806 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:02:31.340106 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 23806
I20250901 20:02:31.340221 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:02:31.340461 14605 external_mini_cluster.cc:1468] Setting key e27d88afb243d3cb1446755461eab229
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756951, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:02:31.348387 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:31.337134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:33027 (local address 127.14.67.126:45747)
0901 20:02:31.337337 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:31.337341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:31.337959 (+ 618us) server_negotiation.cc:407] Connection header received
0901 20:02:31.338822 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:31.338826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:31.338889 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:31.338978 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:31.340153 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.340741 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:31.341848 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.342027 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:31.344584 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:31.344597 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:31.344599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:31.344620 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:31.346255 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:31.346834 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:31.346837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:31.346838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:31.346885 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:31.347294 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:31.347297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:31.347299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:31.347441 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:31.347528 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:31.348166 (+ 638us) server_negotiation.cc:299] Negotiation successful
0901 20:02:31.348257 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250901 20:02:31.349181 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:31.337411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:33027)
0901 20:02:31.337815 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:31.337850 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:31.338631 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:31.339131 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:31.339139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:31.339497 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:31.340007 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:31.340019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.340873 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:31.340875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:31.341725 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:31.341733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.341936 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:31.342534 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:31.342569 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:31.344400 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:31.346391 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:31.346397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:31.346405 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:31.346732 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:31.347026 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:31.347036 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:31.347038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:31.347159 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:31.347575 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:31.347581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:31.347875 (+ 294us) client_negotiation.cc:769] Sending connection context
0901 20:02:31.348089 (+ 214us) client_negotiation.cc:240] Negotiation successful
0901 20:02:31.348350 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":91,"threads_started":1}
I20250901 20:02:31.350421 23935 heartbeater.cc:344] Connected to a master server at 127.14.67.126:45747
I20250901 20:02:31.350703 23935 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:31.351190 23935 heartbeater.cc:507] Master 127.14.67.126:45747 requested a full tablet report, sending...
I20250901 20:02:31.352371 23470 ts_manager.cc:194] Registered new tserver with Master: 9334d040d7e240d886ad5dd74e1a81df (127.14.67.67:41069)
I20250901 20:02:31.353232 23470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:33027
I20250901 20:02:31.354188 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/oryx_127.14.67.65.keytab.
Entry for principal oryx/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/oryx_127.14.67.65.keytab.
WARNING: no policy specified for oryx/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/oryx_127.14.67.66.keytab.
Entry for principal oryx/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/oryx_127.14.67.66.keytab.
WARNING: no policy specified for oryx/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/oryx_127.14.67.67.keytab.
Entry for principal oryx/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/oryx_127.14.67.67.keytab.
I20250901 20:02:31.425346 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 23534
I20250901 20:02:31.432397 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 23670
I20250901 20:02:31.438750 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 23806
I20250901 20:02:31.443961 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 23438
I20250901 20:02:31.450896 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:45747
--webserver_interface=127.14.67.126
--webserver_port=41391
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:45747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:31.549451 23944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:31.549646 23944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:31.549696 23944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:31.552706 23944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:31.552763 23944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:31.552785 23944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:31.552810 23944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:31.552829 23944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:02:31.556720 23944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:45747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:45747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=41391
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:31.557569 23944 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:31.558306 23944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:31.563416 23952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:31.563412 23950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:31.563508 23944 server_base.cc:1047] running on GCE node
W20250901 20:02:31.563431 23949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:31.564112 23944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:31.564888 23944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:31.566038 23944 hybrid_clock.cc:648] HybridClock initialized: now 1756756951566025 us; error 39 us; skew 500 ppm
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756951, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:31.569345 23944 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:02:31.570799 23944 webserver.cc:480] Webserver started at http://127.14.67.126:41391/ using document root <none> and password file <none>
I20250901 20:02:31.571441 23944 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:31.571519 23944 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:31.575724 23944 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:02:31.577698 23959 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:31.578656 23944 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250901 20:02:31.578768 23944 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "c08471d881a44929a3d21048a479af09"
format_stamp: "Formatted at 2025-09-01 20:02:30 on dist-test-slave-2s4d"
server_key: "22715ab0e9446d27eac83250760509ba"
server_key_iv: "3b84a0c65837577904db66045e1ca814"
server_key_version: "encryptionkey@0"
I20250901 20:02:31.579195 23944 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:31.593714 23944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:31.596395 23944 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:31.596603 23944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:31.603533 23944 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:45747
I20250901 20:02:31.603538 24011 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:45747 every 8 connection(s)
I20250901 20:02:31.604578 23944 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:02:31.605880 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 23944
I20250901 20:02:31.606441 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:46539
--local_ip_for_outbound_sockets=127.14.67.65
--tserver_master_addrs=127.14.67.126:45747
--webserver_port=40947
--webserver_interface=127.14.67.65
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.67.65
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/oryx_127.14.67.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250901 20:02:31.608536 24012 sys_catalog.cc:263] Verifying existing consensus state
I20250901 20:02:31.609979 24012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09: Bootstrap starting.
I20250901 20:02:31.625530 24012 log.cc:826] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:31.631290 24012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250901 20:02:31.631613 24012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09: Bootstrap complete.
I20250901 20:02:31.634899 24012 raft_consensus.cc:357] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08471d881a44929a3d21048a479af09" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 45747 } }
I20250901 20:02:31.635192 24012 raft_consensus.cc:738] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08471d881a44929a3d21048a479af09, State: Initialized, Role: FOLLOWER
I20250901 20:02:31.635644 24012 consensus_queue.cc:260] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08471d881a44929a3d21048a479af09" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 45747 } }
I20250901 20:02:31.635776 24012 raft_consensus.cc:397] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:02:31.635841 24012 raft_consensus.cc:491] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:02:31.635951 24012 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 1 FOLLOWER]: Advancing to term 2
I20250901 20:02:31.637373 24012 raft_consensus.cc:513] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08471d881a44929a3d21048a479af09" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 45747 } }
I20250901 20:02:31.637656 24012 leader_election.cc:304] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c08471d881a44929a3d21048a479af09; no voters:
I20250901 20:02:31.637912 24012 leader_election.cc:290] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250901 20:02:31.637988 24016 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 2 FOLLOWER]: Leader election won for term 2
I20250901 20:02:31.638223 24016 raft_consensus.cc:695] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [term 2 LEADER]: Becoming Leader. State: Replica: c08471d881a44929a3d21048a479af09, State: Running, Role: LEADER
I20250901 20:02:31.638639 24016 consensus_queue.cc:237] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08471d881a44929a3d21048a479af09" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 45747 } }
I20250901 20:02:31.639078 24012 sys_catalog.cc:564] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:31.640116 24017 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c08471d881a44929a3d21048a479af09. Latest consensus state: current_term: 2 leader_uuid: "c08471d881a44929a3d21048a479af09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08471d881a44929a3d21048a479af09" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 45747 } } }
I20250901 20:02:31.640306 24017 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:31.640779 24025 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:02:31.640805 24018 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c08471d881a44929a3d21048a479af09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08471d881a44929a3d21048a479af09" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 45747 } } }
I20250901 20:02:31.640936 24018 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:31.643833 24025 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:02:31.644526 24025 catalog_manager.cc:1261] Loaded cluster ID: 579943c107a84c30927b9655715267f1
I20250901 20:02:31.644577 24025 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:02:31.646282 24025 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:02:31.646907 24025 catalog_manager.cc:5966] T 00000000000000000000000000000000 P c08471d881a44929a3d21048a479af09: Loaded TSK: 0
I20250901 20:02:31.647833 24025 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250901 20:02:31.736954 24014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:31.737207 24014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:31.737305 24014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:31.741384 24014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:31.741474 24014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:31.741602 24014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:02:31.746335 24014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/oryx_127.14.67.65.keytab
--principal=oryx/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:46539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=40947
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:45747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:31.747602 24014 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:31.748636 24014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:31.756050 24038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:31.756108 24014 server_base.cc:1047] running on GCE node
W20250901 20:02:31.756029 24041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:31.756289 24039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:31.756722 24014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:31.757318 24014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:31.758533 24014 hybrid_clock.cc:648] HybridClock initialized: now 1756756951758493 us; error 35 us; skew 500 ppm
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756951, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:31.761540 24014 init.cc:377] Logged in from keytab as oryx/127.14.67.65@KRBTEST.COM (short username oryx)
I20250901 20:02:31.762761 24014 webserver.cc:480] Webserver started at http://127.14.67.65:40947/ using document root <none> and password file <none>
I20250901 20:02:31.763334 24014 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:31.763403 24014 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:31.766878 24014 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250901 20:02:31.768709 24048 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:31.769675 24014 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:31.769778 24014 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "6116f165ceda4efabcf6dfc90fe683d1"
format_stamp: "Formatted at 2025-09-01 20:02:30 on dist-test-slave-2s4d"
server_key: "69cc6d34d3c09e7f3ababd698ee4e396"
server_key_iv: "56b053fd1da3c66dfe653b6b6d627583"
server_key_version: "encryptionkey@0"
I20250901 20:02:31.770164 24014 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:31.791491 24014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:31.794399 24014 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:31.794611 24014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:31.795149 24014 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:31.795928 24014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:31.795977 24014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:31.796034 24014 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:31.796075 24014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:31.806993 24014 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:46539
I20250901 20:02:31.807085 24161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:46539 every 8 connection(s)
I20250901 20:02:31.808022 24014 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:02:31.813282 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 24014
I20250901 20:02:31.813843 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:44511
--local_ip_for_outbound_sockets=127.14.67.66
--tserver_master_addrs=127.14.67.126:45747
--webserver_port=44613
--webserver_interface=127.14.67.66
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.67.66
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/oryx_127.14.67.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
W20250901 20:02:31.819242 24165 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:31.810101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:36573 (local address 127.14.67.126:45747)
0901 20:02:31.810629 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:31.810641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:31.810687 (+ 46us) server_negotiation.cc:407] Connection header received
0901 20:02:31.811747 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:31.811762 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:31.812091 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:31.812415 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:31.813509 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.814459 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:31.815132 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.815382 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:31.818722 (+ 3340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:36573: BlockingRecv error: recv got EOF from 127.14.67.65:36573 (error 108)
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":133,"threads_started":1}
I20250901 20:02:31.819239 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:31.810106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:36573)
0901 20:02:31.810565 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:31.810609 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:31.811347 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:31.812461 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:31.812468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:31.812824 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:31.813337 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:31.813349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.814609 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:31.814613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:31.815006 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:31.815013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.815210 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:31.816404 (+ 1194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:31.816428 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:31.818520 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":118,"threads_started":1}
W20250901 20:02:31.819793 24162 heartbeater.cc:646] Failed to heartbeat to 127.14.67.126:45747 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.67.126:45747: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:31.825206 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:31.820207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:44217)
0901 20:02:31.820379 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:31.820394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:31.820646 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:31.821164 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:31.821167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:31.821371 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:31.821592 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:31.821598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.822453 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:31.822456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:31.822875 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:31.822884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.822997 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:31.823555 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:31.823569 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:31.825082 (+ 1513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":66}
W20250901 20:02:31.825321 24165 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:31.820274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:44217 (local address 127.14.67.126:45747)
0901 20:02:31.820901 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:31.820905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:31.820918 (+ 13us) server_negotiation.cc:407] Connection header received
0901 20:02:31.820956 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:31.820958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:31.821009 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:31.821094 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:31.821730 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.822295 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:31.822994 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.823200 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:31.825204 (+ 2004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:44217: BlockingRecv error: recv got EOF from 127.14.67.65:44217 (error 108)
Metrics: {"server-negotiator.queue_time_us":505}
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:31.830865 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:31.825821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:54587)
0901 20:02:31.825972 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:31.825985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:31.826069 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:31.826291 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:31.826294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:31.826433 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:31.826637 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:31.826643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.827288 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:31.827291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:31.827629 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:31.827635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.827731 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:31.828242 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:31.828254 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:31.830743 (+ 2489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20250901 20:02:31.830983 24165 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:31.825899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:54587 (local address 127.14.67.126:45747)
0901 20:02:31.826007 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:31.826014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:31.826029 (+ 15us) server_negotiation.cc:407] Connection header received
0901 20:02:31.826157 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:31.826160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:31.826203 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:31.826268 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:31.826747 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.827188 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:31.827727 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:31.827922 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:31.830867 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:54587: BlockingRecv error: recv got EOF from 127.14.67.65:54587 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20250901 20:02:31.831215 24162 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250901 20:02:31.935671 24166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:31.935860 24166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:31.935915 24166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:31.938856 24166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:31.938910 24166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:31.938987 24166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:02:31.942976 24166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/oryx_127.14.67.66.keytab
--principal=oryx/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:44511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=44613
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:45747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:31.943897 24166 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:31.944679 24166 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:31.951159 24174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:31.951123 24171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:31.951138 24172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:31.951519 24166 server_base.cc:1047] running on GCE node
I20250901 20:02:31.951843 24166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:31.952304 24166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:31.953502 24166 hybrid_clock.cc:648] HybridClock initialized: now 1756756951953507 us; error 52 us; skew 500 ppm
Sep 01 20:02:31 dist-test-slave-2s4d krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756951, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:31.956249 24166 init.cc:377] Logged in from keytab as oryx/127.14.67.66@KRBTEST.COM (short username oryx)
I20250901 20:02:31.957286 24166 webserver.cc:480] Webserver started at http://127.14.67.66:44613/ using document root <none> and password file <none>
I20250901 20:02:31.957773 24166 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:31.957849 24166 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:31.961076 24166 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:31.962862 24181 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:31.963761 24166 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:31.963868 24166 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "8612a8b2186c4fb69ff302ae14e64e89"
format_stamp: "Formatted at 2025-09-01 20:02:31 on dist-test-slave-2s4d"
server_key: "cd025b6ac1a5ef495fe9353eacd30a79"
server_key_iv: "940fcf6065d088a807dab5de0e01cde5"
server_key_version: "encryptionkey@0"
I20250901 20:02:31.964216 24166 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:31.995236 24166 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:31.998342 24166 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:31.998569 24166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:31.999125 24166 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:31.999949 24166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:32.000002 24166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:32.000065 24166 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:32.000106 24166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:32.009145 24166 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:44511
I20250901 20:02:32.009171 24294 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:44511 every 8 connection(s)
I20250901 20:02:32.010149 24166 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:02:32.010766 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 24166
I20250901 20:02:32.011268 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:41069
--local_ip_for_outbound_sockets=127.14.67.67
--tserver_master_addrs=127.14.67.126:45747
--webserver_port=33611
--webserver_interface=127.14.67.67
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.67.67
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/oryx_127.14.67.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
W20250901 20:02:32.020835 24165 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:32.012057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:56847 (local address 127.14.67.126:45747)
0901 20:02:32.012215 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:32.012220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:32.012862 (+ 642us) server_negotiation.cc:407] Connection header received
0901 20:02:32.013713 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:32.013720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:32.013787 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:32.013886 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:32.015461 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.016323 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.017036 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.017276 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.020699 (+ 3423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:56847: BlockingRecv error: recv got EOF from 127.14.67.66:56847 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20250901 20:02:32.021133 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:32.012288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:56847)
0901 20:02:32.012710 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:32.012753 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:32.013518 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:32.014020 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:32.014027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:32.014393 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:32.015204 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.015236 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.016500 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.016504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:32.016919 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.016928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.017085 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.018122 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:32.018144 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:32.020539 (+ 2395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":120,"threads_started":1}
W20250901 20:02:32.021600 24295 heartbeater.cc:646] Failed to heartbeat to 127.14.67.126:45747 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.67.126:45747: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:32.027235 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:32.022017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:47267)
0901 20:02:32.022233 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:32.022244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:32.022307 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:32.022579 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:32.022582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:32.022697 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:32.022860 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.022865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.023994 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.023998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:32.024453 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.024465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.024610 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.025245 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:32.025260 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:32.027156 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156}
W20250901 20:02:32.027537 24165 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:32.022120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:47267 (local address 127.14.67.126:45747)
0901 20:02:32.022256 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:32.022263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:32.022278 (+ 15us) server_negotiation.cc:407] Connection header received
0901 20:02:32.022386 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:32.022390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:32.022434 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:32.022551 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:32.022960 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.023773 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.024598 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.024782 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.027279 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:47267: BlockingRecv error: recv got EOF from 127.14.67.66:47267 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:32.032694 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:32.027648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:41403)
0901 20:02:32.027807 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:32.027819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:32.027898 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:32.028106 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:32.028108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:32.028227 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:32.028364 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.028369 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.029303 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.029306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:32.029701 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.029709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.029837 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.030418 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:32.030439 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:32.032615 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20250901 20:02:32.032843 24295 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20250901 20:02:32.032970 24165 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:32.027706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:41403 (local address 127.14.67.126:45747)
0901 20:02:32.027800 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:32.027805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:32.027870 (+ 65us) server_negotiation.cc:407] Connection header received
0901 20:02:32.027970 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:32.027974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:32.028013 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:32.028096 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:32.028457 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.029152 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.029831 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.029989 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.032731 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:41403: BlockingRecv error: recv got EOF from 127.14.67.66:41403 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20250901 20:02:32.115480 24297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:32.115693 24297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:32.115746 24297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:32.118719 24297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:32.118767 24297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:32.118841 24297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:02:32.122661 24297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/krb5kdc/oryx_127.14.67.67.keytab
--principal=oryx/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:41069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=33611
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:45747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:32.123526 24297 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:32.124262 24297 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:32.130020 24304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:32.130052 24306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:32.130128 24303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:32.130106 24297 server_base.cc:1047] running on GCE node
I20250901 20:02:32.130581 24297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:32.131071 24297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:32.132230 24297 hybrid_clock.cc:648] HybridClock initialized: now 1756756952132215 us; error 28 us; skew 500 ppm
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756952, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:32.134878 24297 init.cc:377] Logged in from keytab as oryx/127.14.67.67@KRBTEST.COM (short username oryx)
I20250901 20:02:32.135824 24297 webserver.cc:480] Webserver started at http://127.14.67.67:33611/ using document root <none> and password file <none>
I20250901 20:02:32.136322 24297 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:32.136389 24297 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:32.139462 24297 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250901 20:02:32.141240 24313 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:32.142153 24297 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:32.142257 24297 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "9334d040d7e240d886ad5dd74e1a81df"
format_stamp: "Formatted at 2025-09-01 20:02:31 on dist-test-slave-2s4d"
server_key: "c857a2859869f9e13e6c5f7e4bc09803"
server_key_iv: "99b459a19cc8ee16a61371a012b75f95"
server_key_version: "encryptionkey@0"
I20250901 20:02:32.142663 24297 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:32.169015 24297 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:32.172399 24297 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:32.172577 24297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:32.173157 24297 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:32.174381 24297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:32.174455 24297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:32.174541 24297 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:32.174568 24297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:32.184330 24297 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:41069
I20250901 20:02:32.184355 24426 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:41069 every 8 connection(s)
I20250901 20:02:32.185343 24297 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:02:32.186724 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 24297
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:32.196211 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:32.187683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.14.67.126:45747)
0901 20:02:32.187878 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:32.187881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:32.187891 (+ 10us) server_negotiation.cc:407] Connection header received
0901 20:02:32.187929 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:32.187931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:32.187971 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:32.188050 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:32.188906 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.189410 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.190629 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.190772 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.192122 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:32.192146 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:32.192158 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:32.192193 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:32.193964 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:32.194560 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:32.194564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:32.194569 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:32.194629 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:32.194915 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:32.194921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:32.194924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:32.195263 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:32.195411 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:32.195852 (+ 441us) server_negotiation.cc:299] Negotiation successful
0901 20:02:32.196083 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
W20250901 20:02:32.197060 24165 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:32.187256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:60363 (local address 127.14.67.126:45747)
0901 20:02:32.187427 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:32.187433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:32.188469 (+ 1036us) server_negotiation.cc:407] Connection header received
0901 20:02:32.189461 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:32.189464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:32.189516 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:32.189585 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:32.191554 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.192008 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.192679 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.192885 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.196994 (+ 4109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:60363: BlockingRecv error: recv got EOF from 127.14.67.67:60363 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20250901 20:02:32.197583 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:32.187718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:60363)
0901 20:02:32.188296 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:32.188345 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:32.189269 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:32.189884 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:32.189896 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:32.190536 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:32.191388 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.191406 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.192151 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.192154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:32.192547 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.192554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.192759 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.194008 (+ 1249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:32.194047 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:32.196877 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":172,"threads_started":1}
W20250901 20:02:32.198019 24427 heartbeater.cc:646] Failed to heartbeat to 127.14.67.126:45747 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.67.126:45747: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:32.203087 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:32.198468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:42921)
0901 20:02:32.198608 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:32.198622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:32.198701 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:32.198916 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:32.198919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:32.199061 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:32.199242 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.199247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.199876 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.199880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:32.200241 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.200248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.200346 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.200816 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:32.200831 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:32.202969 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20250901 20:02:32.203200 24165 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:32.198559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:42921 (local address 127.14.67.126:45747)
0901 20:02:32.198669 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:32.198673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:32.198684 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:02:32.198786 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:32.198789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:32.198825 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:32.198889 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:32.199351 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.199776 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.200345 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.200535 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.203088 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:42921: BlockingRecv error: recv got EOF from 127.14.67.67:42921 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:32.207415 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:32.203673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:42935)
0901 20:02:32.203815 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:32.203831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:32.203902 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:32.204087 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:32.204089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:32.204221 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:32.204375 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.204379 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.204984 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.204987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:32.205342 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.205349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.205432 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.205835 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:32.205846 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:32.207327 (+ 1481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20250901 20:02:32.207564 24165 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:32.203771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:42935 (local address 127.14.67.126:45747)
0901 20:02:32.203888 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:32.203891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:32.203902 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:02:32.203969 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:32.203971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:32.204000 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:32.204068 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:32.204490 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.204905 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.205446 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.205582 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.207439 (+ 1857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:42935: BlockingRecv error: recv got EOF from 127.14.67.67:42935 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20250901 20:02:32.207597 24427 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:32 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:32.837306 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:32.831975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:56519)
0901 20:02:32.832456 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:32.832476 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:32.832623 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:32.832900 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:32.832902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:32.833104 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:32.833304 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.833310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.834064 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.834067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:32.834463 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:32.834470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.834625 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.835353 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:32.835367 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:32.837165 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":128,"threads_started":1}
W20250901 20:02:32.837432 24433 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:32.832112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:56519 (local address 127.14.67.126:45747)
0901 20:02:32.832402 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:32.832406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:32.832642 (+ 236us) server_negotiation.cc:407] Connection header received
0901 20:02:32.832707 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:32.832710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:32.832767 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:32.832863 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:32.833431 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.833937 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:32.834619 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:32.834747 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:32.837289 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:56519: BlockingRecv error: recv got EOF from 127.14.67.65:56519 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
Sep 01 20:02:33 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:33 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:33.038201 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:33.033444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:56251)
0901 20:02:33.033731 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:33.033745 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:33.033842 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:33.034065 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:33.034067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:33.034255 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:33.034458 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:33.034464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:33.035252 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:33.035255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:33.035673 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:33.035679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:33.035771 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:33.036343 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:33.036357 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:33.038075 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":81,"threads_started":1}
W20250901 20:02:33.038336 24433 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:33.033542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:56251 (local address 127.14.67.126:45747)
0901 20:02:33.033671 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:33.033674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:33.033809 (+ 135us) server_negotiation.cc:407] Connection header received
0901 20:02:33.033920 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:33.033922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:33.033966 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:33.034034 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:33.034639 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:33.035148 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:33.035778 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:33.035924 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:33.038229 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:56251: BlockingRecv error: recv got EOF from 127.14.67.66:56251 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 01 20:02:33 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:33 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:33.213727 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:33.208192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:39841)
0901 20:02:33.208454 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:33.208467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:33.208552 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:33.208904 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:33.208906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:33.209079 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:33.209279 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:33.209284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:33.210160 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:33.210164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:33.210637 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:33.210646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:33.210758 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:33.211470 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:33.211485 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:33.213606 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
W20250901 20:02:33.213910 24433 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:33.208351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:39841 (local address 127.14.67.126:45747)
0901 20:02:33.208517 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:33.208521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:33.208536 (+ 15us) server_negotiation.cc:407] Connection header received
0901 20:02:33.208694 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:33.208697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:33.208761 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:33.208849 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:33.209415 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:33.209988 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:33.210783 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:33.210964 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:33.213777 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:39841: BlockingRecv error: recv got EOF from 127.14.67.67:39841 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 01 20:02:33 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:33 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:33.843185 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:33.838181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:40875)
0901 20:02:33.838528 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:33.838544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:33.838743 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:33.839048 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:33.839051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:33.839224 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:33.839415 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:33.839421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:33.840202 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:33.840206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:33.840663 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:33.840671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:33.840767 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:33.841376 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:33.841390 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:33.843065 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":81,"threads_started":1}
W20250901 20:02:33.843524 24437 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:33.838238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:40875 (local address 127.14.67.126:45747)
0901 20:02:33.838553 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:33.838558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:33.838627 (+ 69us) server_negotiation.cc:407] Connection header received
0901 20:02:33.838865 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:33.838869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:33.838929 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:33.839017 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:33.839506 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:33.840080 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:33.840782 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:33.840987 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:33.843353 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:40875: BlockingRecv error: recv got EOF from 127.14.67.65:40875 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
Sep 01 20:02:34 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:34 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:34.044041 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:34.039000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:39755)
0901 20:02:34.039247 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:34.039260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:34.039356 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:34.039726 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:34.039730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:34.039922 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:34.040166 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:34.040174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:34.040883 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:34.040886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:34.041322 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:34.041330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:34.041419 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:34.041954 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:34.041966 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:34.043924 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
W20250901 20:02:34.044351 24437 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:34.039103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:39755 (local address 127.14.67.126:45747)
0901 20:02:34.039226 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:34.039230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:34.039345 (+ 115us) server_negotiation.cc:407] Connection header received
0901 20:02:34.039417 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:34.039420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:34.039464 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:34.039533 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:34.040283 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:34.040772 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:34.041417 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:34.041544 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:34.044194 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:39755: BlockingRecv error: recv got EOF from 127.14.67.66:39755 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 01 20:02:34 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:34 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:34.219668 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:34.214447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:45691)
0901 20:02:34.214717 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:34.214730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:34.214820 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:34.215134 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:34.215138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:34.215366 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:34.215619 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:34.215626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:34.216494 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:34.216497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:34.216907 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:34.216913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:34.217010 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:34.217593 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:34.217606 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:34.219524 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
W20250901 20:02:34.219762 24437 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:34.214593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:45691 (local address 127.14.67.126:45747)
0901 20:02:34.214729 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:34.214732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:34.214794 (+ 62us) server_negotiation.cc:407] Connection header received
0901 20:02:34.214897 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:34.214899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:34.214940 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:34.215005 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:34.215859 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:34.216363 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:34.217053 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:34.217210 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:34.219652 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:45691: BlockingRecv error: recv got EOF from 127.14.67.67:45691 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 01 20:02:34 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:34 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:34.849959 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:34.844133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:33381)
0901 20:02:34.844452 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:34.844467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:34.844572 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:34.844959 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:34.844962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:34.845172 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:34.845421 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:34.845428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:34.846422 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:34.846425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:34.846956 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:34.846965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:34.847068 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:34.847705 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:34.847718 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:34.849753 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
W20250901 20:02:34.850035 24441 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:34.844244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:33381 (local address 127.14.67.126:45747)
0901 20:02:34.844500 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:34.844503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:34.844519 (+ 16us) server_negotiation.cc:407] Connection header received
0901 20:02:34.844714 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:34.844717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:34.844775 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:34.844879 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:34.845667 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:34.846267 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:34.847092 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:34.847238 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:34.849905 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:33381: BlockingRecv error: recv got EOF from 127.14.67.65:33381 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
Sep 01 20:02:35 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:35 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:35.050987 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:35.045087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:47225)
0901 20:02:35.045411 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:35.045426 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:35.045565 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:35.045847 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:35.045850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:35.046143 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:35.046373 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:35.046380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:35.047211 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:35.047217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:35.047612 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:35.047619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:35.047740 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:35.048461 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:35.048482 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:35.050796 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":107,"threads_started":1}
W20250901 20:02:35.051079 24441 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:35.045167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:47225 (local address 127.14.67.126:45747)
0901 20:02:35.045357 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:35.045362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:35.045508 (+ 146us) server_negotiation.cc:407] Connection header received
0901 20:02:35.045662 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:35.045665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:35.045728 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:35.045800 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:35.046545 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:35.047102 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:35.047733 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:35.047961 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:35.050954 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:47225: BlockingRecv error: recv got EOF from 127.14.67.66:47225 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 01 20:02:35 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:35 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:35.226078 24443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:35.220463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:39475)
0901 20:02:35.220765 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:35.220779 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:35.220890 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:35.221156 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:35.221159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:35.221328 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:35.221529 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:35.221534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:35.222446 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:35.222452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:35.222895 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:35.222903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:35.223033 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:35.223676 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:35.223696 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:35.225912 (+ 2216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":112,"threads_started":1}
W20250901 20:02:35.226151 24441 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:35.220575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:39475 (local address 127.14.67.126:45747)
0901 20:02:35.220721 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:35.220727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:35.220850 (+ 123us) server_negotiation.cc:407] Connection header received
0901 20:02:35.220969 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:35.220972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:35.221024 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:35.221101 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:35.221658 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:35.222324 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:35.223005 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:35.223199 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:35.226039 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:39475: BlockingRecv error: recv got EOF from 127.14.67.67:39475 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 01 20:02:35 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:35 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:35.859977 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:35.850894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:47253)
0901 20:02:35.851224 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:35.851239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:35.851329 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:35.851642 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:35.851645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:35.851832 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:35.852051 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:35.852057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:35.852918 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:35.852920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:35.853323 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:35.853330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:35.853427 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:35.857523 (+ 4096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:35.857541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:35.859764 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":95,"threads_started":1}
W20250901 20:02:35.860247 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:35.851003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:47253 (local address 127.14.67.126:45747)
0901 20:02:35.851339 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:35.851343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:35.851382 (+ 39us) server_negotiation.cc:407] Connection header received
0901 20:02:35.851422 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:35.851425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:35.851479 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:35.851563 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:35.852202 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:35.852808 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:35.853470 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:35.853647 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:35.860089 (+ 6442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:47253: BlockingRecv error: recv got EOF from 127.14.67.65:47253 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
Sep 01 20:02:36 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:36 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:36.057145 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:36.052202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:43065)
0901 20:02:36.052565 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:36.052579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:36.052709 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:36.053076 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:36.053080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:36.053283 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:36.053474 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:36.053480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:36.054205 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:36.054211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:36.054614 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:36.054620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:36.054725 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:36.055351 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:36.055363 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:36.056966 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":126,"threads_started":1}
W20250901 20:02:36.057186 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:36.052255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:43065 (local address 127.14.67.126:45747)
0901 20:02:36.052439 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:36.052443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:36.052665 (+ 222us) server_negotiation.cc:407] Connection header received
0901 20:02:36.052806 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:36.052811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:36.052916 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:36.053038 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:36.053599 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:36.054104 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:36.054726 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:36.054922 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:36.057091 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:43065: BlockingRecv error: recv got EOF from 127.14.67.66:43065 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 01 20:02:36 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:36 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:36.232540 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:36.227038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:55741)
0901 20:02:36.227359 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:36.227374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:36.227499 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:36.227785 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:36.227788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:36.228022 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:36.228258 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:36.228265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:36.229021 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:36.229024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:36.229449 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:36.229456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:36.229558 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:36.230215 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:36.230228 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:36.232402 (+ 2174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":97,"threads_started":1}
W20250901 20:02:36.232863 24445 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:36.227211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:55741 (local address 127.14.67.126:45747)
0901 20:02:36.227382 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:36.227386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:36.227491 (+ 105us) server_negotiation.cc:407] Connection header received
0901 20:02:36.227579 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:36.227581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:36.227636 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:36.227714 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:36.228382 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:36.228888 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:36.229559 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:36.229686 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:36.232590 (+ 2904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:55741: BlockingRecv error: recv got EOF from 127.14.67.67:55741 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 01 20:02:36 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:36 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:36.866639 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:36.860896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:51523)
0901 20:02:36.861176 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:36.861189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:36.861268 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:36.861499 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:36.861501 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:36.861670 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:36.861872 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:36.861877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:36.863100 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:36.863104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:36.863518 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:36.863529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:36.863675 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:36.864371 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:36.864386 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:36.866420 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
W20250901 20:02:36.866829 24449 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:36.861017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:51523 (local address 127.14.67.126:45747)
0901 20:02:36.861267 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:36.861271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:36.861283 (+ 12us) server_negotiation.cc:407] Connection header received
0901 20:02:36.861357 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:36.861359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:36.861402 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:36.861478 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:36.862076 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:36.862918 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:36.863681 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:36.863840 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:36.866632 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:51523: BlockingRecv error: recv got EOF from 127.14.67.65:51523 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
Sep 01 20:02:37 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:37 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:37.062863 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:37.057946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:54621)
0901 20:02:37.058211 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:37.058224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:37.058318 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:37.058649 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:37.058652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:37.058840 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:37.059032 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:37.059038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:37.059748 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:37.059754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:37.060133 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:37.060139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:37.060242 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:37.060826 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:37.060838 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:37.062710 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
W20250901 20:02:37.062958 24449 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:37.058261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:54621 (local address 127.14.67.126:45747)
0901 20:02:37.058436 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:37.058440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:37.058450 (+ 10us) server_negotiation.cc:407] Connection header received
0901 20:02:37.058477 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:37.058479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:37.058544 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:37.058613 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:37.059157 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:37.059644 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:37.060243 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:37.060450 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:37.062842 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:54621: BlockingRecv error: recv got EOF from 127.14.67.66:54621 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 01 20:02:37 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:37 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:37.239770 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:37.233618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:52717)
0901 20:02:37.233949 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:37.233968 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:37.234102 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:37.234573 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:37.234576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:37.234775 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:37.234985 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:37.234991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:37.236056 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:37.236061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:37.236676 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:37.236688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:37.236826 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:37.237457 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:37.237471 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:37.239567 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":111,"threads_started":1}
W20250901 20:02:37.239778 24449 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:37.234090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:52717 (local address 127.14.67.126:45747)
0901 20:02:37.234262 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:37.234268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:37.234282 (+ 14us) server_negotiation.cc:407] Connection header received
0901 20:02:37.234329 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:37.234333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:37.234390 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:37.234485 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:37.235135 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:37.235912 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:37.236823 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:37.237050 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:37.239686 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:52717: BlockingRecv error: recv got EOF from 127.14.67.67:52717 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 01 20:02:37 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:37 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:37.873620 24452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:37.867652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:40301)
0901 20:02:37.868028 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:37.868043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:37.868169 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:37.868517 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:37.868520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:37.868773 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:37.869050 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:37.869069 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:37.869898 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:37.869903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:37.870339 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:37.870346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:37.870462 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:37.871167 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:37.871181 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:37.873475 (+ 2294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
W20250901 20:02:37.873780 24453 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:37.867721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:40301 (local address 127.14.67.126:45747)
0901 20:02:37.868038 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:37.868045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:37.868132 (+ 87us) server_negotiation.cc:407] Connection header received
0901 20:02:37.868260 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:37.868263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:37.868315 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:37.868424 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:37.869212 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:37.869751 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:37.870486 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:37.870673 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:37.873634 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:40301: BlockingRecv error: recv got EOF from 127.14.67.65:40301 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":87,"threads_started":1}
Sep 01 20:02:38 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:38 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:38.069514 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:38.063811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:44503)
0901 20:02:38.064186 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:38.064201 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:38.064362 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:38.064683 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:38.064687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:38.064885 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:38.065091 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:38.065097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:38.065967 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:38.065971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:38.066417 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:38.066424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:38.066572 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:38.067294 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:38.067307 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:38.069387 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":103,"threads_started":1}
W20250901 20:02:38.069742 24453 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:38.063937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:44503 (local address 127.14.67.126:45747)
0901 20:02:38.064124 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:38.064131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:38.064343 (+ 212us) server_negotiation.cc:407] Connection header received
0901 20:02:38.064456 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:38.064458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:38.064523 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:38.064602 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:38.065213 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:38.065795 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:38.066625 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:38.066798 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:38.069583 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:44503: BlockingRecv error: recv got EOF from 127.14.67.66:44503 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 01 20:02:38 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:38 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:38.245625 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:38.240529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:59003)
0901 20:02:38.240779 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:38.240792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:38.240919 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:38.241239 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:38.241241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:38.241435 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:38.241631 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:38.241637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:38.242399 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:38.242402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:38.242849 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:38.242858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:38.242959 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:38.243548 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:38.243570 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:38.245511 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
W20250901 20:02:38.245746 24453 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:38.240585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:59003 (local address 127.14.67.126:45747)
0901 20:02:38.240724 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:38.240729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:38.240892 (+ 163us) server_negotiation.cc:407] Connection header received
0901 20:02:38.241025 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:38.241029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:38.241080 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:38.241182 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:38.241774 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:38.242275 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:38.242974 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:38.243112 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:38.245636 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:59003: BlockingRecv error: recv got EOF from 127.14.67.67:59003 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 01 20:02:38 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:38 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:38.879948 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:38.874812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:33919)
0901 20:02:38.875320 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:38.875335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:38.875464 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:38.875733 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:38.875736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:38.875933 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:38.876129 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:38.876135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:38.876947 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:38.876950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:38.877324 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:38.877330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:38.877434 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:38.878006 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:38.878018 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:38.879831 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":94,"threads_started":1}
W20250901 20:02:38.880115 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:38.875034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:33919 (local address 127.14.67.126:45747)
0901 20:02:38.875302 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:38.875306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:38.875412 (+ 106us) server_negotiation.cc:407] Connection header received
0901 20:02:38.875569 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:38.875573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:38.875621 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:38.875697 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:38.876239 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:38.876839 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:38.877423 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:38.877614 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:38.879979 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:33919: BlockingRecv error: recv got EOF from 127.14.67.65:33919 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":66,"threads_started":1}
Sep 01 20:02:39 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:39 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:39.075841 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:39.070449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:59599)
0901 20:02:39.070921 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:39.070938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:39.071068 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:39.071410 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:39.071413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:39.071638 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:39.071910 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:39.071920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:39.072889 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:39.072892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:39.073334 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:39.073342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:39.073460 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:39.074031 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:39.074043 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:39.075715 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":144,"threads_started":1}
W20250901 20:02:39.075943 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:39.070561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:59599 (local address 127.14.67.126:45747)
0901 20:02:39.070712 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:39.070715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:39.071037 (+ 322us) server_negotiation.cc:407] Connection header received
0901 20:02:39.071183 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:39.071187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:39.071266 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:39.071377 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:39.072055 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:39.072736 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:39.073484 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:39.073650 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:39.075846 (+ 2196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:59599: BlockingRecv error: recv got EOF from 127.14.67.66:59599 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 01 20:02:39 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:39 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:39.251227 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:39.246496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:57557)
0901 20:02:39.246891 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:39.246904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:39.247026 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:39.247291 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:39.247294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:39.247482 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:39.247664 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:39.247669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:39.248378 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:39.248381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:39.248740 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:39.248746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:39.248849 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:39.249468 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:39.249480 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:39.251069 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":117,"threads_started":1}
W20250901 20:02:39.251286 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:39.246644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:57557 (local address 127.14.67.126:45747)
0901 20:02:39.246816 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:39.246820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:39.246971 (+ 151us) server_negotiation.cc:407] Connection header received
0901 20:02:39.247132 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:39.247138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:39.247186 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:39.247254 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:39.247778 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:39.248271 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:39.248839 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:39.249025 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:39.251185 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:57557: BlockingRecv error: recv got EOF from 127.14.67.67:57557 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 01 20:02:39 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:39 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:39.885712 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:39.880727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:38341)
0901 20:02:39.880992 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:39.881004 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:39.881088 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:39.881313 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:39.881315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:39.881474 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:39.881661 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:39.881666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:39.882410 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:39.882413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:39.882896 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:39.882905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:39.883004 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:39.883577 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:39.883590 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:39.885602 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":94,"threads_started":1}
W20250901 20:02:39.886075 24461 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:39.880870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:38341 (local address 127.14.67.126:45747)
0901 20:02:39.881096 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:39.881099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:39.881138 (+ 39us) server_negotiation.cc:407] Connection header received
0901 20:02:39.881173 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:39.881176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:39.881220 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:39.881294 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:39.881778 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:39.882306 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:39.883036 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:39.883237 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:39.885904 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:38341: BlockingRecv error: recv got EOF from 127.14.67.65:38341 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
Sep 01 20:02:40 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:40 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:40.082149 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:40.076819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:46881)
0901 20:02:40.077091 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:40.077104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:40.077202 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:40.077472 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:40.077474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:40.077640 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:40.077835 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:40.077840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:40.079205 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:40.079207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:40.079579 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:40.079585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:40.079672 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:40.080323 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:40.080336 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:40.082035 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
W20250901 20:02:40.082319 24461 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:40.076914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:46881 (local address 127.14.67.126:45747)
0901 20:02:40.077074 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:40.077079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:40.077184 (+ 105us) server_negotiation.cc:407] Connection header received
0901 20:02:40.077278 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:40.077281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:40.077328 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:40.077402 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:40.078082 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:40.079087 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:40.079752 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:40.079977 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:40.082175 (+ 2198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:46881: BlockingRecv error: recv got EOF from 127.14.67.66:46881 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 01 20:02:40 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:40 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:40.257165 24463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:40.252237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:51079)
0901 20:02:40.252541 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:40.252555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:40.252680 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:40.252990 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:40.252994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:40.253167 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:40.253363 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:40.253369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:40.254068 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:40.254072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:40.254473 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:40.254481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:40.254592 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:40.255206 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:40.255218 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:40.257010 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
W20250901 20:02:40.257247 24461 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:40.252364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:51079 (local address 127.14.67.126:45747)
0901 20:02:40.252546 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:40.252550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:40.252672 (+ 122us) server_negotiation.cc:407] Connection header received
0901 20:02:40.252759 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:40.252761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:40.252820 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:40.252896 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:40.253464 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:40.253952 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:40.254825 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:40.254968 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:40.257135 (+ 2167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:51079: BlockingRecv error: recv got EOF from 127.14.67.67:51079 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 01 20:02:40 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:40 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:40.897197 24464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:40.886686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:56265)
0901 20:02:40.887033 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:40.887051 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:40.887171 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:40.887521 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:40.887524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:40.887768 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:40.888000 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:40.888006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:40.888998 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:40.889001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:40.889408 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:40.889415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:40.889523 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:40.894948 (+ 5425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:40.894968 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:40.897079 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":92,"threads_started":1}
W20250901 20:02:40.897351 24465 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:40.886746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:56265 (local address 127.14.67.126:45747)
0901 20:02:40.887035 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:40.887039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:40.887146 (+ 107us) server_negotiation.cc:407] Connection header received
0901 20:02:40.887295 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:40.887298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:40.887354 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:40.887459 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:40.888140 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:40.888862 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:40.889555 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:40.889759 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:40.897225 (+ 7466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:56265: BlockingRecv error: recv got EOF from 127.14.67.65:56265 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
Sep 01 20:02:41 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:41 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:41.088027 24466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:41.082896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:56943)
0901 20:02:41.083180 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:41.083193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:41.083291 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:41.083495 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:41.083497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:41.083683 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:41.083875 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:41.083881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:41.084543 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:41.084545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:41.084915 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:41.084922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:41.085009 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:41.085761 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:41.085778 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:41.087876 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
W20250901 20:02:41.088228 24465 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:41.082950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:56943 (local address 127.14.67.126:45747)
0901 20:02:41.083088 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:41.083092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:41.083261 (+ 169us) server_negotiation.cc:407] Connection header received
0901 20:02:41.083367 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:41.083369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:41.083415 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:41.083472 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:41.083961 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:41.084452 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:41.085106 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:41.085395 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:41.088074 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:56943: BlockingRecv error: recv got EOF from 127.14.67.66:56943 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 01 20:02:41 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:41 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:41.262312 24467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:41.257836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:50091)
0901 20:02:41.258107 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:41.258120 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:41.258213 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:41.258422 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:41.258424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:41.258605 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:41.258798 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:41.258803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:41.259520 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:41.259523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:41.259880 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:41.259886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:41.259967 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:41.260500 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:41.260513 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:41.262212 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
W20250901 20:02:41.262583 24465 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:41.257918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:50091 (local address 127.14.67.126:45747)
0901 20:02:41.258073 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:41.258077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:41.258181 (+ 104us) server_negotiation.cc:407] Connection header received
0901 20:02:41.258282 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:41.258285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:41.258326 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:41.258390 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:41.258928 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:41.259428 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:41.259997 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:41.260168 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:41.262347 (+ 2179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:50091: BlockingRecv error: recv got EOF from 127.14.67.67:50091 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 01 20:02:41 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:41 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:41.903009 24469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:41.898135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:39159)
0901 20:02:41.898490 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:41.898533 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:41.898648 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:41.898912 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:41.898915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:41.899123 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:41.899313 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:41.899319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:41.900041 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:41.900044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:41.900412 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:41.900417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:41.900522 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:41.901206 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:41.901218 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:41.902833 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":158,"threads_started":1}
W20250901 20:02:41.903064 24468 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:41.898160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:39159 (local address 127.14.67.126:45747)
0901 20:02:41.898398 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:41.898403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:41.898606 (+ 203us) server_negotiation.cc:407] Connection header received
0901 20:02:41.898736 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:41.898741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:41.898797 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:41.898873 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:41.899423 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:41.899932 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:41.900512 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:41.900709 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:41.902952 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:39159: BlockingRecv error: recv got EOF from 127.14.67.65:39159 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
Sep 01 20:02:42 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:42 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:42.094933 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:42.089066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:41367)
0901 20:02:42.089388 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:42.089403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:42.089514 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:42.089876 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:42.089879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:42.090089 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:42.090310 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:42.090316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:42.091154 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:42.091157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:42.091576 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:42.091584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:42.091686 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:42.092315 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:42.092328 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:42.094806 (+ 2478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":114,"threads_started":1}
W20250901 20:02:42.095103 24468 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:42.089193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:41367 (local address 127.14.67.126:45747)
0901 20:02:42.089395 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:42.089399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:42.089492 (+ 93us) server_negotiation.cc:407] Connection header received
0901 20:02:42.089674 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:42.089678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:42.089734 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:42.089816 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:42.090449 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:42.091038 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:42.091704 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:42.091858 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:42.094971 (+ 3113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:41367: BlockingRecv error: recv got EOF from 127.14.67.66:41367 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 01 20:02:42 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:42 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:42.268378 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:42.263047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:60811)
0901 20:02:42.263303 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:42.263318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:42.263404 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:42.263609 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:42.263611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:42.263786 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:42.263975 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:42.263980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:42.264869 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:42.264872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:42.265249 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:42.265256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:42.265348 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:42.265893 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:42.265905 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:42.268272 (+ 2367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":85,"threads_started":1}
W20250901 20:02:42.268632 24468 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:42.263162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:60811 (local address 127.14.67.126:45747)
0901 20:02:42.263308 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:42.263312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:42.263382 (+ 70us) server_negotiation.cc:407] Connection header received
0901 20:02:42.263479 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:42.263482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:42.263524 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:42.263590 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:42.264162 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:42.264739 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:42.265381 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:42.265526 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:42.268451 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:60811: BlockingRecv error: recv got EOF from 127.14.67.67:60811 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 01 20:02:42 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:42 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:42.909943 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:42.904280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:40929)
0901 20:02:42.904606 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:42.904620 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:42.904712 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:42.905099 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:42.905103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:42.905322 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:42.905565 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:42.905570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:42.906459 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:42.906463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:42.907005 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:42.907019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:42.907159 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:42.908003 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:42.908019 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:42.909826 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":119,"threads_started":1}
W20250901 20:02:42.910233 24473 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:42.904353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:40929 (local address 127.14.67.126:45747)
0901 20:02:42.904654 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:42.904658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:42.904675 (+ 17us) server_negotiation.cc:407] Connection header received
0901 20:02:42.904855 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:42.904858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:42.904917 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:42.905020 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:42.905738 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:42.906321 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:42.907154 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:42.907354 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:42.910066 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:40929: BlockingRecv error: recv got EOF from 127.14.67.65:40929 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
Sep 01 20:02:43 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:43 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:43.100237 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:43.095760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:40819)
0901 20:02:43.096028 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:43.096041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:43.096124 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:43.096421 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:43.096424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:43.096595 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:43.096788 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:43.096793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:43.097468 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:43.097471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:43.097846 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:43.097852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:43.097935 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:43.098485 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:43.098498 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:43.100123 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
W20250901 20:02:43.100359 24473 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:43.095873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:40819 (local address 127.14.67.126:45747)
0901 20:02:43.096071 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:43.096075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:43.096090 (+ 15us) server_negotiation.cc:407] Connection header received
0901 20:02:43.096245 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:43.096254 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:43.096309 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:43.096379 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:43.096878 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:43.097375 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:43.097981 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:43.098117 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:43.100247 (+ 2130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:40819: BlockingRecv error: recv got EOF from 127.14.67.66:40819 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 01 20:02:43 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:43 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:43.274767 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:43.269204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:48307)
0901 20:02:43.269501 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:43.269515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:43.269694 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:43.270045 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:43.270048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:43.270254 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:43.270476 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:43.270482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:43.271494 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:43.271498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:43.271927 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:43.271935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:43.272029 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:43.272722 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:43.272749 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:43.274645 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":90,"threads_started":1}
W20250901 20:02:43.274948 24473 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:43.269316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:48307 (local address 127.14.67.126:45747)
0901 20:02:43.269507 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:43.269511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:43.269648 (+ 137us) server_negotiation.cc:407] Connection header received
0901 20:02:43.269856 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:43.269858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:43.269914 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:43.269992 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:43.270697 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:43.271349 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:43.272047 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:43.272183 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:43.274827 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:48307: BlockingRecv error: recv got EOF from 127.14.67.67:48307 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 01 20:02:43 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:43 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:43.916082 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:43.910789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:43703)
0901 20:02:43.911130 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:43.911146 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:43.911274 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:43.911594 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:43.911597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:43.911832 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:43.912092 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:43.912099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:43.912865 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:43.912876 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:43.913316 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:43.913323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:43.913426 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:43.914220 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:43.914235 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:43.915969 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":90,"threads_started":1}
W20250901 20:02:43.916267 24477 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:43.910933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:43703 (local address 127.14.67.126:45747)
0901 20:02:43.911162 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:43.911165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:43.911229 (+ 64us) server_negotiation.cc:407] Connection header received
0901 20:02:43.911358 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:43.911360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:43.911406 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:43.911491 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:43.912201 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:43.912747 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:43.913414 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:43.913541 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:43.916116 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:43703: BlockingRecv error: recv got EOF from 127.14.67.65:43703 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
Sep 01 20:02:44 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:44 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:44.105538 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:44.101032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:34729)
0901 20:02:44.101281 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:44.101294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:44.101392 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:44.101602 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:44.101604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:44.101785 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:44.101975 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:44.101980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:44.102711 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:44.102713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:44.103076 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:44.103082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:44.103171 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:44.103749 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:44.103769 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:44.105434 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
W20250901 20:02:44.105665 24477 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:44.101146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:34729 (local address 127.14.67.126:45747)
0901 20:02:44.101293 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:44.101296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:44.101360 (+ 64us) server_negotiation.cc:407] Connection header received
0901 20:02:44.101463 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:44.101465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:44.101509 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:44.101579 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:44.102063 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:44.102599 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:44.103187 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:44.103346 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:44.105557 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:34729: BlockingRecv error: recv got EOF from 127.14.67.66:34729 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 01 20:02:44 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:44 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:44.280234 24479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:44.275485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:42469)
0901 20:02:44.275769 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:44.275781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:44.275889 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:44.276132 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:44.276135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:44.276329 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:44.276516 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:44.276521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:44.277218 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:44.277221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:44.277587 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:44.277593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:44.277694 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:44.278262 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:44.278277 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:44.280129 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
W20250901 20:02:44.280381 24477 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:44.275571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:42469 (local address 127.14.67.126:45747)
0901 20:02:44.275712 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:44.275716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:44.275846 (+ 130us) server_negotiation.cc:407] Connection header received
0901 20:02:44.275988 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:44.275991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:44.276034 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:44.276104 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:44.276630 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:44.277115 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:44.277690 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:44.277891 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:44.280269 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:42469: BlockingRecv error: recv got EOF from 127.14.67.67:42469 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 01 20:02:44 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:44 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:44.922302 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:44.917071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:40289)
0901 20:02:44.917417 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:44.917433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:44.917558 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:44.917908 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:44.917920 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:44.918104 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:44.918315 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:44.918322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:44.919214 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:44.919217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:44.919672 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:44.919682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:44.919806 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:44.920391 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:44.920405 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:44.922147 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
W20250901 20:02:44.922416 24481 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:44.917176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:40289 (local address 127.14.67.126:45747)
0901 20:02:44.917450 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:44.917454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:44.917521 (+ 67us) server_negotiation.cc:407] Connection header received
0901 20:02:44.917658 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:44.917661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:44.917711 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:44.917816 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:44.918454 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:44.919104 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:44.919840 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:44.920044 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:44.922290 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:40289: BlockingRecv error: recv got EOF from 127.14.67.65:40289 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
Sep 01 20:02:45 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:45 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:45.112046 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:45.106428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:39855)
0901 20:02:45.106833 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:45.106851 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:45.106992 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:45.107376 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:45.107379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:45.107638 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:45.107858 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:45.107864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:45.108624 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:45.108627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:45.109015 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:45.109021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:45.109129 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:45.109808 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:45.109822 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:45.111908 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":174,"threads_started":1}
W20250901 20:02:45.112227 24481 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:45.106634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:39855 (local address 127.14.67.126:45747)
0901 20:02:45.106814 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:45.106820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:45.106982 (+ 162us) server_negotiation.cc:407] Connection header received
0901 20:02:45.107200 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:45.107203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:45.107267 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:45.107339 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:45.107978 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:45.108509 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:45.109120 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:45.109315 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:45.112062 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:39855: BlockingRecv error: recv got EOF from 127.14.67.66:39855 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 01 20:02:45 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:45 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:45.285784 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:45.281107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:55885)
0901 20:02:45.281464 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:45.281479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:45.281589 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:45.281813 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:45.281815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:45.281988 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:45.282183 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:45.282188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:45.282961 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:45.282964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:45.283405 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:45.283412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:45.283497 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:45.284040 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:45.284052 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:45.285662 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":132,"threads_started":1}
W20250901 20:02:45.285915 24481 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:45.281166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:55885 (local address 127.14.67.126:45747)
0901 20:02:45.281340 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:45.281359 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:45.281585 (+ 226us) server_negotiation.cc:407] Connection header received
0901 20:02:45.281662 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:45.281665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:45.281718 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:45.281789 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:45.282274 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:45.282841 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:45.283501 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:45.283624 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:45.285799 (+ 2175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:55885: BlockingRecv error: recv got EOF from 127.14.67.67:55885 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 01 20:02:45 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:45 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:45.931903 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:45.923147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:38219)
0901 20:02:45.923429 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:45.923442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:45.923530 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:45.923740 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:45.923742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:45.923912 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:45.924102 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:45.924107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:45.925099 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:45.925102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:45.925500 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:45.925506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:45.925599 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:45.929412 (+ 3813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:45.929430 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:45.931795 (+ 2365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
W20250901 20:02:45.932093 24485 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:45.923264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:38219 (local address 127.14.67.126:45747)
0901 20:02:45.923549 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:45.923552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:45.923563 (+ 11us) server_negotiation.cc:407] Connection header received
0901 20:02:45.923594 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:45.923597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:45.923638 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:45.923713 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:45.924307 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:45.924953 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:45.925630 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:45.925814 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:45.931939 (+ 6125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:38219: BlockingRecv error: recv got EOF from 127.14.67.65:38219 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
Sep 01 20:02:46 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:46 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:46.118687 24486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:46.112920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:59385)
0901 20:02:46.113235 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:46.113250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:46.113373 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:46.113731 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:46.113734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:46.113922 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:46.114146 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:46.114153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:46.115109 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:46.115112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:46.115595 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:46.115603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:46.115713 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:46.116313 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:46.116331 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:46.118565 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
W20250901 20:02:46.118811 24485 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:46.113028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:59385 (local address 127.14.67.126:45747)
0901 20:02:46.113165 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:46.113168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:46.113372 (+ 204us) server_negotiation.cc:407] Connection header received
0901 20:02:46.113447 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:46.113450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:46.113513 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:46.113607 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:46.114309 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:46.114985 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:46.115739 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:46.115926 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:46.118704 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:59385: BlockingRecv error: recv got EOF from 127.14.67.66:59385 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 01 20:02:46 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:46 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.67@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:46.292173 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:46.286677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.67:59099)
0901 20:02:46.287023 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:46.287037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:46.287147 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:46.287439 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:46.287441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:46.287614 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:46.287826 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:46.287831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:46.288571 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:46.288574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:46.288955 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:46.288961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:46.289052 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:46.289607 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:46.289619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:46.291995 (+ 2376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":146,"threads_started":1}
W20250901 20:02:46.292423 24485 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:46.286737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:59099 (local address 127.14.67.126:45747)
0901 20:02:46.286959 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:46.286975 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:46.287122 (+ 147us) server_negotiation.cc:407] Connection header received
0901 20:02:46.287247 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:46.287250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:46.287322 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:46.287405 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:46.287953 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:46.288468 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:46.289073 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:46.289198 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:46.292154 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.67:59099: BlockingRecv error: recv got EOF from 127.14.67.67:59099 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 01 20:02:46 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:46 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.65@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:46.937793 24488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:46.932808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.65:52381)
0901 20:02:46.933139 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:46.933154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:46.933261 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:46.933547 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:46.933550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:46.933781 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:46.934004 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:46.934010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:46.934893 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:46.934898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:46.935305 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:46.935314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:46.935424 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:46.936038 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:46.936051 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:46.937665 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
W20250901 20:02:46.937960 24489 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:46.932918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:52381 (local address 127.14.67.126:45747)
0901 20:02:46.933183 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:46.933187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:46.933216 (+ 29us) server_negotiation.cc:407] Connection header received
0901 20:02:46.933344 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:46.933346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:46.933396 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:46.933482 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:46.934177 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:46.934750 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:46.935407 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:46.935527 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:46.937824 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.65:52381: BlockingRecv error: recv got EOF from 127.14.67.65:52381 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
Sep 01 20:02:47 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
Sep 01 20:02:47 dist-test-slave-2s4d krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.67.66@KRBTEST.COM for oryx/127.14.67.126@KRBTEST.COM, Server not found in Kerberos database
I20250901 20:02:47.124909 24490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:47.119550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:45747 (local address 127.14.67.66:45629)
0901 20:02:47.119831 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:47.119848 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:47.119965 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:47.120210 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:47.120213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:47.120429 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:47.120625 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:47.120631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.121339 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:47.121345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:47.121726 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:47.121732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.121842 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:47.122469 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:47.122482 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:47.124736 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.67.126:45747: Server oryx/127.14.67.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
W20250901 20:02:47.124965 24489 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:47.119661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:45629 (local address 127.14.67.126:45747)
0901 20:02:47.119831 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:47.119840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:47.119930 (+ 90us) server_negotiation.cc:407] Connection header received
0901 20:02:47.120061 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:47.120065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:47.120115 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:47.120185 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:47.120747 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.121237 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:47.121840 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.122047 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:47.124870 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.67.66:45629: BlockingRecv error: recv got EOF from 127.14.67.66:45629 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20250901 20:02:47.199954 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 24014
I20250901 20:02:47.205382 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 24166
I20250901 20:02:47.210850 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 24297
I20250901 20:02:47.216215 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 23944
2025-09-01T20:02:47Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (19080 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-09-01T20:02:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:02:47Z Disabled control of system clock
I20250901 20:02:47.248801 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:46259
--webserver_interface=127.14.67.126
--webserver_port=0
--builtin_ntp_servers=127.14.67.84:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:46259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20250901 20:02:47.364523 24494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:47.364774 24494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:47.364818 24494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:47.367949 24494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:47.368005 24494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:47.368022 24494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:47.368042 24494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:47.368059 24494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:02:47.371729 24494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:46259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:46259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:47.372954 24494 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:47.374233 24494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:47.380839 24500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:47.380964 24502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:47.380828 24499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:47.381081 24494 server_base.cc:1047] running on GCE node
I20250901 20:02:47.381861 24494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:47.383074 24494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:47.384248 24494 hybrid_clock.cc:648] HybridClock initialized: now 1756756967384241 us; error 32 us; skew 500 ppm
I20250901 20:02:47.386160 24494 webserver.cc:480] Webserver started at http://127.14.67.126:38657/ using document root <none> and password file <none>
I20250901 20:02:47.386792 24494 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:47.386859 24494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:47.387043 24494 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:47.388767 24494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "bd2bdb89767b4786905688d22e04e135"
format_stamp: "Formatted at 2025-09-01 20:02:47 on dist-test-slave-2s4d"
server_key: "e111c3162fb58700028543ca56af5ed7"
server_key_iv: "63c4a6fdc9d484830c9089a639277fb4"
server_key_version: "encryptionkey@0"
I20250901 20:02:47.389236 24494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "bd2bdb89767b4786905688d22e04e135"
format_stamp: "Formatted at 2025-09-01 20:02:47 on dist-test-slave-2s4d"
server_key: "e111c3162fb58700028543ca56af5ed7"
server_key_iv: "63c4a6fdc9d484830c9089a639277fb4"
server_key_version: "encryptionkey@0"
I20250901 20:02:47.392691 24494 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:02:47.394893 24508 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:47.395845 24494 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:47.395948 24494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "bd2bdb89767b4786905688d22e04e135"
format_stamp: "Formatted at 2025-09-01 20:02:47 on dist-test-slave-2s4d"
server_key: "e111c3162fb58700028543ca56af5ed7"
server_key_iv: "63c4a6fdc9d484830c9089a639277fb4"
server_key_version: "encryptionkey@0"
I20250901 20:02:47.396029 24494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:47.405959 24494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:47.406486 24494 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:47.406673 24494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:47.413563 24494 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:46259
I20250901 20:02:47.413563 24560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:46259 every 8 connection(s)
I20250901 20:02:47.414611 24494 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:02:47.416987 24561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:47.422065 24561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135: Bootstrap starting.
I20250901 20:02:47.424290 24561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:47.424839 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 24494
I20250901 20:02:47.424938 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:02:47.425011 24561 log.cc:826] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:47.425220 14605 external_mini_cluster.cc:1468] Setting key cb3be93c059fad2a28af69e07c8574fd
I20250901 20:02:47.426996 24561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135: No bootstrap required, opened a new log
I20250901 20:02:47.429862 24561 raft_consensus.cc:357] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2bdb89767b4786905688d22e04e135" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46259 } }
I20250901 20:02:47.430068 24561 raft_consensus.cc:383] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:47.430128 24561 raft_consensus.cc:738] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2bdb89767b4786905688d22e04e135, State: Initialized, Role: FOLLOWER
I20250901 20:02:47.430585 24561 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2bdb89767b4786905688d22e04e135" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46259 } }
I20250901 20:02:47.430701 24561 raft_consensus.cc:397] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:02:47.430761 24561 raft_consensus.cc:491] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:02:47.430860 24561 raft_consensus.cc:3058] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:47.431829 24561 raft_consensus.cc:513] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2bdb89767b4786905688d22e04e135" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46259 } }
I20250901 20:02:47.432179 24561 leader_election.cc:304] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd2bdb89767b4786905688d22e04e135; no voters:
I20250901 20:02:47.432492 24561 leader_election.cc:290] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:02:47.432610 24566 raft_consensus.cc:2802] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:47.432861 24566 raft_consensus.cc:695] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [term 1 LEADER]: Becoming Leader. State: Replica: bd2bdb89767b4786905688d22e04e135, State: Running, Role: LEADER
I20250901 20:02:47.433166 24566 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2bdb89767b4786905688d22e04e135" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46259 } }
I20250901 20:02:47.433596 24561 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:47.433646 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:47.426250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.14.67.126:46259)
0901 20:02:47.426915 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:47.426926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:47.426958 (+ 32us) server_negotiation.cc:407] Connection header received
0901 20:02:47.427613 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:47.427634 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:47.427977 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:47.428289 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:47.428902 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.430042 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:47.430967 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.431272 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:47.431772 (+ 500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:47.431795 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:47.431808 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:02:47.432246 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:47.432294 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:02:47.432311 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:47.432428 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:47.432602 (+ 174us) server_negotiation.cc:299] Negotiation successful
0901 20:02:47.432972 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":134,"threads_started":1}
I20250901 20:02:47.435483 24567 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd2bdb89767b4786905688d22e04e135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2bdb89767b4786905688d22e04e135" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46259 } } }
I20250901 20:02:47.435602 24568 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd2bdb89767b4786905688d22e04e135. Latest consensus state: current_term: 1 leader_uuid: "bd2bdb89767b4786905688d22e04e135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2bdb89767b4786905688d22e04e135" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46259 } } }
I20250901 20:02:47.436206 24568 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:47.436203 24567 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135 [sys.catalog]: This master's current role is: LEADER
W20250901 20:02:47.437661 24580 catalog_manager.cc:1560] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250901 20:02:47.437722 24580 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250901 20:02:47.437791 24581 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:02:47.438449 24581 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:02:47.442742 24581 catalog_manager.cc:1349] Generated new cluster ID: 984603fb727f4dbea65c9278e51e774e
I20250901 20:02:47.442855 24581 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:02:47.459797 24581 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:02:47.460642 24581 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:02:47.471715 24581 catalog_manager.cc:5955] T 00000000000000000000000000000000 P bd2bdb89767b4786905688d22e04e135: Generated new TSK 0
I20250901 20:02:47.472355 24581 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 20:02:47.477942 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:46259
--builtin_ntp_servers=127.14.67.84:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20250901 20:02:47.597041 24585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:47.597314 24585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:47.597394 24585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:47.601106 24585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:47.601171 24585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:47.601279 24585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:02:47.605223 24585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:46259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:47.606483 24585 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:47.607851 24585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:47.615103 24590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:47.615070 24593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:47.615347 24585 server_base.cc:1047] running on GCE node
W20250901 20:02:47.615070 24591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:47.615813 24585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:47.616411 24585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:47.617580 24585 hybrid_clock.cc:648] HybridClock initialized: now 1756756967617564 us; error 28 us; skew 500 ppm
I20250901 20:02:47.619341 24585 webserver.cc:480] Webserver started at http://127.14.67.65:36785/ using document root <none> and password file <none>
I20250901 20:02:47.619894 24585 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:47.619963 24585 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:47.620141 24585 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:47.621883 24585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "94cd4a9e9d5e4dbca1687fa01291e28c"
format_stamp: "Formatted at 2025-09-01 20:02:47 on dist-test-slave-2s4d"
server_key: "0fdca4e6b5c0ba7de28d8b16b7d65e4b"
server_key_iv: "815e1d4de7bf3c948dfc21a270022edc"
server_key_version: "encryptionkey@0"
I20250901 20:02:47.622345 24585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "94cd4a9e9d5e4dbca1687fa01291e28c"
format_stamp: "Formatted at 2025-09-01 20:02:47 on dist-test-slave-2s4d"
server_key: "0fdca4e6b5c0ba7de28d8b16b7d65e4b"
server_key_iv: "815e1d4de7bf3c948dfc21a270022edc"
server_key_version: "encryptionkey@0"
I20250901 20:02:47.625495 24585 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250901 20:02:47.627877 24599 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:47.629174 24585 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250901 20:02:47.629307 24585 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "94cd4a9e9d5e4dbca1687fa01291e28c"
format_stamp: "Formatted at 2025-09-01 20:02:47 on dist-test-slave-2s4d"
server_key: "0fdca4e6b5c0ba7de28d8b16b7d65e4b"
server_key_iv: "815e1d4de7bf3c948dfc21a270022edc"
server_key_version: "encryptionkey@0"
I20250901 20:02:47.629424 24585 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:47.642765 24585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:47.643349 24585 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:47.643507 24585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:47.644023 24585 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:47.645146 24585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:47.645223 24585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:47.645298 24585 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:47.645336 24585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:47.656211 24585 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:44655
I20250901 20:02:47.656236 24712 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:44655 every 8 connection(s)
I20250901 20:02:47.657095 24585 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:02:47.663897 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 24585
I20250901 20:02:47.664160 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:02:47.664592 14605 external_mini_cluster.cc:1468] Setting key 25f68ecc9fea9057c8a7a13c9dfc7461
I20250901 20:02:47.665035 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:47.658801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:40985 (local address 127.14.67.126:46259)
0901 20:02:47.658939 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:47.658943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:47.659771 (+ 828us) server_negotiation.cc:407] Connection header received
0901 20:02:47.660536 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:47.660539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:47.660596 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:47.660666 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:47.661537 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.662040 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:47.662700 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.662840 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:47.664270 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:47.664306 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:47.664310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:02:47.664335 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:47.664370 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:02:47.664384 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:47.664527 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:47.664687 (+ 160us) server_negotiation.cc:299] Negotiation successful
0901 20:02:47.664791 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20250901 20:02:47.665547 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:47.659075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46259 (local address 127.14.67.65:40985)
0901 20:02:47.659649 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:47.659681 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:47.660372 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:47.660805 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:47.660812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:47.660881 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:47.661419 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:47.661430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.662172 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:47.662176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:47.662598 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:47.662605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.662770 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:47.663941 (+ 1171us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:02:47.663943 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:47.663973 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:02:47.664034 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:47.664530 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:47.664537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:47.664541 (+ 4us) client_negotiation.cc:769] Sending connection context
0901 20:02:47.664673 (+ 132us) client_negotiation.cc:240] Negotiation successful
0901 20:02:47.664874 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
I20250901 20:02:47.666867 24713 heartbeater.cc:344] Connected to a master server at 127.14.67.126:46259
I20250901 20:02:47.667093 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:46259
--builtin_ntp_servers=127.14.67.84:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20250901 20:02:47.667133 24713 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:47.667744 24713 heartbeater.cc:507] Master 127.14.67.126:46259 requested a full tablet report, sending...
I20250901 20:02:47.669275 24524 ts_manager.cc:194] Registered new tserver with Master: 94cd4a9e9d5e4dbca1687fa01291e28c (127.14.67.65:44655)
I20250901 20:02:47.670418 24524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.67.65:40985
W20250901 20:02:47.775614 24716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:47.775879 24716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:47.775964 24716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:47.780287 24716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:47.780360 24716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:47.780479 24716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:02:47.784284 24716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:46259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:47.785388 24716 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:47.786490 24716 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:47.793051 24721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:47.793063 24724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:47.793380 24716 server_base.cc:1047] running on GCE node
W20250901 20:02:47.793078 24722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:47.793730 24716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:47.794333 24716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:47.795480 24716 hybrid_clock.cc:648] HybridClock initialized: now 1756756967795458 us; error 40 us; skew 500 ppm
I20250901 20:02:47.797062 24716 webserver.cc:480] Webserver started at http://127.14.67.66:46123/ using document root <none> and password file <none>
I20250901 20:02:47.797552 24716 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:47.797603 24716 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:47.797748 24716 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:47.799345 24716 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "b34f4e9f1e5944fe9b824238c62bd04a"
format_stamp: "Formatted at 2025-09-01 20:02:47 on dist-test-slave-2s4d"
server_key: "19b0e45a80a4d64bf6708b150c536e85"
server_key_iv: "c119ce33d0727bbd0c5382d34b51f8af"
server_key_version: "encryptionkey@0"
I20250901 20:02:47.799757 24716 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "b34f4e9f1e5944fe9b824238c62bd04a"
format_stamp: "Formatted at 2025-09-01 20:02:47 on dist-test-slave-2s4d"
server_key: "19b0e45a80a4d64bf6708b150c536e85"
server_key_iv: "c119ce33d0727bbd0c5382d34b51f8af"
server_key_version: "encryptionkey@0"
I20250901 20:02:47.802790 24716 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250901 20:02:47.804900 24730 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:47.806146 24716 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:47.806303 24716 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "b34f4e9f1e5944fe9b824238c62bd04a"
format_stamp: "Formatted at 2025-09-01 20:02:47 on dist-test-slave-2s4d"
server_key: "19b0e45a80a4d64bf6708b150c536e85"
server_key_iv: "c119ce33d0727bbd0c5382d34b51f8af"
server_key_version: "encryptionkey@0"
I20250901 20:02:47.806413 24716 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:47.825074 24716 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:47.825608 24716 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:47.825773 24716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:47.826785 24716 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:47.828023 24716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:47.828083 24716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:47.828150 24716 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:47.828177 24716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:47.838449 24716 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:38659
I20250901 20:02:47.838476 24843 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:38659 every 8 connection(s)
I20250901 20:02:47.839313 24716 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:02:47.842739 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 24716
I20250901 20:02:47.842869 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:02:47.843161 14605 external_mini_cluster.cc:1468] Setting key 339ace70aa8efc61dc5aa13f267944af
I20250901 20:02:47.845218 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:46259
--builtin_ntp_servers=127.14.67.84:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20250901 20:02:47.847397 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:47.840975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:50241 (local address 127.14.67.126:46259)
0901 20:02:47.841108 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:47.841111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:47.841793 (+ 682us) server_negotiation.cc:407] Connection header received
0901 20:02:47.842617 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:47.842620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:47.842667 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:47.842745 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:47.843827 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.844306 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:47.845090 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.845283 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:47.846635 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:47.846662 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:47.846664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:02:47.846682 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:47.846706 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:02:47.846714 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:47.846834 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:47.847063 (+ 229us) server_negotiation.cc:299] Negotiation successful
0901 20:02:47.847172 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20250901 20:02:47.847995 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:47.841134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46259 (local address 127.14.67.66:50241)
0901 20:02:47.841675 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:47.841708 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:47.842416 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:47.842913 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:47.842925 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:47.843024 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:47.843687 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:47.843701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.844425 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:47.844428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:47.844899 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:47.844908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:47.845115 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:47.846351 (+ 1236us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:02:47.846355 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:47.846395 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:02:47.846460 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:47.846843 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:47.846853 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:47.846861 (+ 8us) client_negotiation.cc:769] Sending connection context
0901 20:02:47.847026 (+ 165us) client_negotiation.cc:240] Negotiation successful
0901 20:02:47.847217 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":98,"threads_started":1}
I20250901 20:02:47.849243 24844 heartbeater.cc:344] Connected to a master server at 127.14.67.126:46259
I20250901 20:02:47.849599 24844 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:47.850492 24844 heartbeater.cc:507] Master 127.14.67.126:46259 requested a full tablet report, sending...
I20250901 20:02:47.851903 24524 ts_manager.cc:194] Registered new tserver with Master: b34f4e9f1e5944fe9b824238c62bd04a (127.14.67.66:38659)
I20250901 20:02:47.852552 24524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.67.66:50241
W20250901 20:02:47.952471 24847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:47.952672 24847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:47.952714 24847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:47.955724 24847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:47.955771 24847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:47.955835 24847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:02:47.959278 24847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:46259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:47.960191 24847 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:47.961195 24847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:47.967343 24855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:47.967367 24853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:47.967348 24852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:47.967531 24847 server_base.cc:1047] running on GCE node
I20250901 20:02:47.967836 24847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:47.968302 24847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:47.969476 24847 hybrid_clock.cc:648] HybridClock initialized: now 1756756967969463 us; error 56 us; skew 500 ppm
I20250901 20:02:47.971166 24847 webserver.cc:480] Webserver started at http://127.14.67.67:46477/ using document root <none> and password file <none>
I20250901 20:02:47.971743 24847 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:47.971801 24847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:47.971974 24847 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:47.973522 24847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "ad94a006429544458df359ff9bd6ef72"
format_stamp: "Formatted at 2025-09-01 20:02:47 on dist-test-slave-2s4d"
server_key: "71c7449176376a168637564d10742da0"
server_key_iv: "f9c813061c904fec1471efda98c75bcf"
server_key_version: "encryptionkey@0"
I20250901 20:02:47.973963 24847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "ad94a006429544458df359ff9bd6ef72"
format_stamp: "Formatted at 2025-09-01 20:02:47 on dist-test-slave-2s4d"
server_key: "71c7449176376a168637564d10742da0"
server_key_iv: "f9c813061c904fec1471efda98c75bcf"
server_key_version: "encryptionkey@0"
I20250901 20:02:47.976910 24847 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250901 20:02:47.978905 24861 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:47.980036 24847 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:47.980140 24847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "ad94a006429544458df359ff9bd6ef72"
format_stamp: "Formatted at 2025-09-01 20:02:47 on dist-test-slave-2s4d"
server_key: "71c7449176376a168637564d10742da0"
server_key_iv: "f9c813061c904fec1471efda98c75bcf"
server_key_version: "encryptionkey@0"
I20250901 20:02:47.980230 24847 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:47.992301 24847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:47.992780 24847 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:47.992910 24847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:47.993345 24847 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:47.994155 24847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:47.994202 24847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:47.994237 24847 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:47.994257 24847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:48.002539 24847 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:38275
I20250901 20:02:48.002584 24974 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:38275 every 8 connection(s)
I20250901 20:02:48.003320 24847 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:02:48.010155 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:48.004740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:52605 (local address 127.14.67.126:46259)
0901 20:02:48.004852 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:48.004855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:48.005596 (+ 741us) server_negotiation.cc:407] Connection header received
0901 20:02:48.006349 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:48.006352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:48.006411 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:48.006482 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:48.007352 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:48.007827 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:48.008426 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:48.008600 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:48.009694 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:48.009709 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:48.009712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:02:48.009722 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:48.009740 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:02:48.009747 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:48.009808 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:48.009960 (+ 152us) server_negotiation.cc:299] Negotiation successful
0901 20:02:48.010014 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20250901 20:02:48.010727 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:48.004962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46259 (local address 127.14.67.67:52605)
0901 20:02:48.005472 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:48.005507 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:48.006170 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:48.006643 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:48.006650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:48.006716 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:48.007216 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:48.007226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:48.007947 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:48.007950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:48.008323 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:48.008329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:48.008493 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:48.009494 (+ 1001us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:02:48.009495 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:48.009520 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:02:48.009564 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:48.009819 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:48.009829 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:48.009835 (+ 6us) client_negotiation.cc:769] Sending connection context
0901 20:02:48.009955 (+ 120us) client_negotiation.cc:240] Negotiation successful
0901 20:02:48.010137 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
I20250901 20:02:48.011698 24975 heartbeater.cc:344] Connected to a master server at 127.14.67.126:46259
I20250901 20:02:48.011888 24975 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:48.012143 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 24847
I20250901 20:02:48.012255 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:02:48.012372 24975 heartbeater.cc:507] Master 127.14.67.126:46259 requested a full tablet report, sending...
I20250901 20:02:48.012471 14605 external_mini_cluster.cc:1468] Setting key 5bed6ebb5c1d403cac1d7c673a5e078a
I20250901 20:02:48.013535 24524 ts_manager.cc:194] Registered new tserver with Master: ad94a006429544458df359ff9bd6ef72 (127.14.67.67:38275)
I20250901 20:02:48.014441 24524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.67.67:52605
I20250901 20:02:48.014983 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250901 20:02:48.017645 24565 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:48.016982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.14.67.126:46259)
0901 20:02:48.017113 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:48.017117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:48.017205 (+ 88us) server_negotiation.cc:407] Connection header received
0901 20:02:48.017293 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:48.017295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:48.017328 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:48.017392 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:48.017550 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40124: BlockingRecv error: recv got EOF from 127.0.0.1:40124 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20250901 20:02:48.018069 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 24585
I20250901 20:02:48.024466 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 24716
I20250901 20:02:48.029426 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 24847
I20250901 20:02:48.034686 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 24494
2025-09-01T20:02:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (815 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-09-01T20:02:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:02:48Z Disabled control of system clock
I20250901 20:02:48.062863 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:39283
--webserver_interface=127.14.67.126
--webserver_port=0
--builtin_ntp_servers=127.14.67.84:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:39283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20250901 20:02:48.160244 24986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:48.160418 24986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:48.160461 24986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:48.163437 24986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:48.163488 24986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:48.163504 24986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:48.163523 24986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:48.163544 24986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:02:48.167068 24986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:39283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:39283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:48.167917 24986 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:48.168870 24986 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:48.174741 24994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:48.174799 24991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:48.174799 24992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:48.174978 24986 server_base.cc:1047] running on GCE node
I20250901 20:02:48.175400 24986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:48.176173 24986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:48.177330 24986 hybrid_clock.cc:648] HybridClock initialized: now 1756756968177317 us; error 39 us; skew 500 ppm
I20250901 20:02:48.178910 24986 webserver.cc:480] Webserver started at http://127.14.67.126:43945/ using document root <none> and password file <none>
I20250901 20:02:48.179651 24986 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:48.179749 24986 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:48.180066 24986 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:48.181661 24986 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "2ce32fd6284a46458956641722978fdd"
format_stamp: "Formatted at 2025-09-01 20:02:48 on dist-test-slave-2s4d"
server_key: "b0778adecb1d97d3794c2a6243925418"
server_key_iv: "9b638cc5dcd411845f7481ba54c6301b"
server_key_version: "encryptionkey@0"
I20250901 20:02:48.182088 24986 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "2ce32fd6284a46458956641722978fdd"
format_stamp: "Formatted at 2025-09-01 20:02:48 on dist-test-slave-2s4d"
server_key: "b0778adecb1d97d3794c2a6243925418"
server_key_iv: "9b638cc5dcd411845f7481ba54c6301b"
server_key_version: "encryptionkey@0"
I20250901 20:02:48.185277 24986 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:02:48.187299 25000 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:48.188508 24986 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:48.188699 24986 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "2ce32fd6284a46458956641722978fdd"
format_stamp: "Formatted at 2025-09-01 20:02:48 on dist-test-slave-2s4d"
server_key: "b0778adecb1d97d3794c2a6243925418"
server_key_iv: "9b638cc5dcd411845f7481ba54c6301b"
server_key_version: "encryptionkey@0"
I20250901 20:02:48.188820 24986 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:48.194128 24986 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:48.194824 24986 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:48.194991 24986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:48.202440 25052 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:39283 every 8 connection(s)
I20250901 20:02:48.202441 24986 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:39283
I20250901 20:02:48.203408 24986 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:02:48.206404 25053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:48.207072 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 24986
I20250901 20:02:48.207221 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:02:48.207530 14605 external_mini_cluster.cc:1468] Setting key 9a5da0f4e137bdf95366004869b87e32
I20250901 20:02:48.212950 25053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd: Bootstrap starting.
I20250901 20:02:48.213649 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:48.209133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49510 (local address 127.14.67.126:39283)
0901 20:02:48.210153 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:48.210163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:48.210195 (+ 32us) server_negotiation.cc:407] Connection header received
0901 20:02:48.210799 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:48.210816 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:48.211122 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:48.211386 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:48.212396 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:48.212402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:48.212413 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:02:48.212426 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:48.212476 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:02:48.212490 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:48.212640 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:48.212739 (+ 99us) server_negotiation.cc:299] Negotiation successful
0901 20:02:48.212993 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":169,"threads_started":1}
I20250901 20:02:48.215598 25053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:48.216455 25053 log.cc:826] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:48.218112 25053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd: No bootstrap required, opened a new log
I20250901 20:02:48.220530 25053 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce32fd6284a46458956641722978fdd" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39283 } }
I20250901 20:02:48.220748 25053 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:48.220811 25053 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce32fd6284a46458956641722978fdd, State: Initialized, Role: FOLLOWER
I20250901 20:02:48.221207 25053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce32fd6284a46458956641722978fdd" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39283 } }
I20250901 20:02:48.221326 25053 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:02:48.221385 25053 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:02:48.221483 25053 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:48.222306 25053 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce32fd6284a46458956641722978fdd" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39283 } }
I20250901 20:02:48.222622 25053 leader_election.cc:304] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ce32fd6284a46458956641722978fdd; no voters:
I20250901 20:02:48.222872 25053 leader_election.cc:290] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:02:48.222956 25058 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:48.223174 25058 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [term 1 LEADER]: Becoming Leader. State: Replica: 2ce32fd6284a46458956641722978fdd, State: Running, Role: LEADER
I20250901 20:02:48.223426 25058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce32fd6284a46458956641722978fdd" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39283 } }
I20250901 20:02:48.223865 25053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:48.224843 25059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ce32fd6284a46458956641722978fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce32fd6284a46458956641722978fdd" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39283 } } }
I20250901 20:02:48.224833 25060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ce32fd6284a46458956641722978fdd. Latest consensus state: current_term: 1 leader_uuid: "2ce32fd6284a46458956641722978fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce32fd6284a46458956641722978fdd" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 39283 } } }
I20250901 20:02:48.224979 25060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:48.224979 25059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:48.225603 25067 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:02:48.229996 25067 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:02:48.235321 25067 catalog_manager.cc:1349] Generated new cluster ID: 39a03a5f8082415492278ba9bd313ee1
I20250901 20:02:48.235385 25067 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:02:48.247577 25067 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:02:48.247822 25067 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:02:48.258405 25067 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2ce32fd6284a46458956641722978fdd: Generated new TSK 0
I20250901 20:02:48.259109 25067 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 20:02:48.271096 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:39283
--builtin_ntp_servers=127.14.67.84:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20250901 20:02:48.368752 25077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:48.368927 25077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:48.368974 25077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:48.371989 25077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:48.372033 25077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:48.372110 25077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:02:48.375777 25077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:39283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:48.376657 25077 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:48.377954 25077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:48.383786 25085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:48.383803 25082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:48.383816 25083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:48.384275 25077 server_base.cc:1047] running on GCE node
I20250901 20:02:48.384568 25077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:48.385023 25077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:48.386260 25077 hybrid_clock.cc:648] HybridClock initialized: now 1756756968386228 us; error 33 us; skew 500 ppm
I20250901 20:02:48.388445 25077 webserver.cc:480] Webserver started at http://127.14.67.65:44923/ using document root <none> and password file <none>
I20250901 20:02:48.389000 25077 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:48.389060 25077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:48.389223 25077 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:48.390830 25077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "b36ca8a493d94fcf912ce580d5b9869c"
format_stamp: "Formatted at 2025-09-01 20:02:48 on dist-test-slave-2s4d"
server_key: "db5e0aaaaecadd480a75d324a8226dcc"
server_key_iv: "3a01cac4f388bfe067a7656be00d91bd"
server_key_version: "encryptionkey@0"
I20250901 20:02:48.391268 25077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "b36ca8a493d94fcf912ce580d5b9869c"
format_stamp: "Formatted at 2025-09-01 20:02:48 on dist-test-slave-2s4d"
server_key: "db5e0aaaaecadd480a75d324a8226dcc"
server_key_iv: "3a01cac4f388bfe067a7656be00d91bd"
server_key_version: "encryptionkey@0"
I20250901 20:02:48.394238 25077 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250901 20:02:48.396276 25091 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:48.397245 25077 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:48.397343 25077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "b36ca8a493d94fcf912ce580d5b9869c"
format_stamp: "Formatted at 2025-09-01 20:02:48 on dist-test-slave-2s4d"
server_key: "db5e0aaaaecadd480a75d324a8226dcc"
server_key_iv: "3a01cac4f388bfe067a7656be00d91bd"
server_key_version: "encryptionkey@0"
I20250901 20:02:48.397418 25077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:48.401914 25077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:48.402421 25077 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:48.402581 25077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:48.403038 25077 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:48.403825 25077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:48.403869 25077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:48.403923 25077 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:48.403949 25077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:48.413082 25077 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:33769
I20250901 20:02:48.413146 25204 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:33769 every 8 connection(s)
I20250901 20:02:48.413894 25077 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:02:48.415692 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 25077
I20250901 20:02:48.415805 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:02:48.416051 14605 external_mini_cluster.cc:1468] Setting key f174208084e0f762205ff90e820847e6
I20250901 20:02:48.417703 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:39283
--builtin_ntp_servers=127.14.67.84:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20250901 20:02:48.419277 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:48.415399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:59903 (local address 127.14.67.126:39283)
0901 20:02:48.415551 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:48.415556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:48.416807 (+ 1251us) server_negotiation.cc:407] Connection header received
0901 20:02:48.417720 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:48.417723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:48.417764 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:48.417841 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:48.418860 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:48.418864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:48.418866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:02:48.418878 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:48.418898 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:02:48.418906 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:48.418979 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:48.419115 (+ 136us) server_negotiation.cc:299] Negotiation successful
0901 20:02:48.419155 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20250901 20:02:48.419885 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:48.415676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39283 (local address 127.14.67.65:59903)
0901 20:02:48.416688 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:48.416720 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:48.417531 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:48.417981 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:48.417990 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:48.418062 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:48.418643 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:02:48.418645 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:48.418672 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:02:48.418720 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:48.418976 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:48.418984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:48.418990 (+ 6us) client_negotiation.cc:769] Sending connection context
0901 20:02:48.419148 (+ 158us) client_negotiation.cc:240] Negotiation successful
0901 20:02:48.419333 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":119,"threads_started":1}
I20250901 20:02:48.421072 25205 heartbeater.cc:344] Connected to a master server at 127.14.67.126:39283
I20250901 20:02:48.421263 25205 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:48.421597 25205 heartbeater.cc:507] Master 127.14.67.126:39283 requested a full tablet report, sending...
I20250901 20:02:48.422866 25017 ts_manager.cc:194] Registered new tserver with Master: b36ca8a493d94fcf912ce580d5b9869c (127.14.67.65:33769)
W20250901 20:02:48.515110 25208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:48.515295 25208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:48.515372 25208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:48.518311 25208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:48.518358 25208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:48.518458 25208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:02:48.521991 25208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:39283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:48.522945 25208 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:48.523960 25208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:48.530376 25213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:48.530421 25214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:48.530424 25216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:48.530707 25208 server_base.cc:1047] running on GCE node
I20250901 20:02:48.531159 25208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:48.531855 25208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:48.533013 25208 hybrid_clock.cc:648] HybridClock initialized: now 1756756968532997 us; error 35 us; skew 500 ppm
I20250901 20:02:48.534785 25208 webserver.cc:480] Webserver started at http://127.14.67.66:44753/ using document root <none> and password file <none>
I20250901 20:02:48.535275 25208 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:48.535328 25208 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:48.535488 25208 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:48.537048 25208 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "ed49a877639e4b6fa792ae4a57c57d82"
format_stamp: "Formatted at 2025-09-01 20:02:48 on dist-test-slave-2s4d"
server_key: "c6950fc3755cd17bcd881f2a5602b9fd"
server_key_iv: "6511e8ab10040f9e87ff97c2d373315d"
server_key_version: "encryptionkey@0"
I20250901 20:02:48.537474 25208 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "ed49a877639e4b6fa792ae4a57c57d82"
format_stamp: "Formatted at 2025-09-01 20:02:48 on dist-test-slave-2s4d"
server_key: "c6950fc3755cd17bcd881f2a5602b9fd"
server_key_iv: "6511e8ab10040f9e87ff97c2d373315d"
server_key_version: "encryptionkey@0"
I20250901 20:02:48.540635 25208 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250901 20:02:48.542718 25222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:48.543603 25208 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:48.543725 25208 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "ed49a877639e4b6fa792ae4a57c57d82"
format_stamp: "Formatted at 2025-09-01 20:02:48 on dist-test-slave-2s4d"
server_key: "c6950fc3755cd17bcd881f2a5602b9fd"
server_key_iv: "6511e8ab10040f9e87ff97c2d373315d"
server_key_version: "encryptionkey@0"
I20250901 20:02:48.543802 25208 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:48.548085 25208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:48.548554 25208 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:48.548692 25208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:48.549130 25208 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:48.549911 25208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:48.549957 25208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:48.550004 25208 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:48.550033 25208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:48.558099 25208 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:36861
I20250901 20:02:48.558143 25335 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:36861 every 8 connection(s)
I20250901 20:02:48.558976 25208 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:02:48.562583 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 25208
I20250901 20:02:48.562711 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:02:48.562956 14605 external_mini_cluster.cc:1468] Setting key ecbf25e95f76fb51e7a235007c2893d7
I20250901 20:02:48.564059 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:48.560872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:56983 (local address 127.14.67.126:39283)
0901 20:02:48.561017 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:48.561021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:48.561787 (+ 766us) server_negotiation.cc:407] Connection header received
0901 20:02:48.562569 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:48.562572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:48.562612 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:48.562696 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:48.563684 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:48.563687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:48.563690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:02:48.563700 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:48.563720 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:02:48.563729 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:48.563807 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:48.563897 (+ 90us) server_negotiation.cc:299] Negotiation successful
0901 20:02:48.563933 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250901 20:02:48.564644 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:39283
--builtin_ntp_servers=127.14.67.84:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20250901 20:02:48.564725 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:48.561115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39283 (local address 127.14.67.66:56983)
0901 20:02:48.561662 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:48.561693 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:48.562342 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:48.562847 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:48.562854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:48.562933 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:48.563470 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:02:48.563471 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:48.563498 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:02:48.563545 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:48.563798 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:48.563803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:48.563805 (+ 2us) client_negotiation.cc:769] Sending connection context
0901 20:02:48.563924 (+ 119us) client_negotiation.cc:240] Negotiation successful
0901 20:02:48.564100 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
I20250901 20:02:48.565827 25336 heartbeater.cc:344] Connected to a master server at 127.14.67.126:39283
I20250901 20:02:48.566071 25336 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:48.566375 25336 heartbeater.cc:507] Master 127.14.67.126:39283 requested a full tablet report, sending...
I20250901 20:02:48.567286 25017 ts_manager.cc:194] Registered new tserver with Master: ed49a877639e4b6fa792ae4a57c57d82 (127.14.67.66:36861)
W20250901 20:02:48.672230 25339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:48.672451 25339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:48.672501 25339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:48.675591 25339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:48.675643 25339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:48.675716 25339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:02:48.679306 25339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:39283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:48.680510 25339 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:48.681839 25339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:48.688889 25345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:48.688889 25347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:48.688987 25344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:48.689272 25339 server_base.cc:1047] running on GCE node
I20250901 20:02:48.689616 25339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:48.690210 25339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:48.691372 25339 hybrid_clock.cc:648] HybridClock initialized: now 1756756968691360 us; error 25 us; skew 500 ppm
I20250901 20:02:48.693087 25339 webserver.cc:480] Webserver started at http://127.14.67.67:37993/ using document root <none> and password file <none>
I20250901 20:02:48.693607 25339 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:48.693671 25339 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:48.693843 25339 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:48.695546 25339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "31b835f05b33430d893101426c80ab11"
format_stamp: "Formatted at 2025-09-01 20:02:48 on dist-test-slave-2s4d"
server_key: "61d28631675358f8dcdc51c783b43ced"
server_key_iv: "2c51ef08d5837e48e00ccdbe24f49f4a"
server_key_version: "encryptionkey@0"
I20250901 20:02:48.695971 25339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "31b835f05b33430d893101426c80ab11"
format_stamp: "Formatted at 2025-09-01 20:02:48 on dist-test-slave-2s4d"
server_key: "61d28631675358f8dcdc51c783b43ced"
server_key_iv: "2c51ef08d5837e48e00ccdbe24f49f4a"
server_key_version: "encryptionkey@0"
I20250901 20:02:48.699138 25339 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250901 20:02:48.701085 25353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:48.701997 25339 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:48.702090 25339 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "31b835f05b33430d893101426c80ab11"
format_stamp: "Formatted at 2025-09-01 20:02:48 on dist-test-slave-2s4d"
server_key: "61d28631675358f8dcdc51c783b43ced"
server_key_iv: "2c51ef08d5837e48e00ccdbe24f49f4a"
server_key_version: "encryptionkey@0"
I20250901 20:02:48.702165 25339 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:48.706394 25339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:48.706911 25339 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:48.707056 25339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:48.707511 25339 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:48.708310 25339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:48.708356 25339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:48.708410 25339 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:48.708436 25339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:48.718659 25339 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:36629
I20250901 20:02:48.718699 25466 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:36629 every 8 connection(s)
I20250901 20:02:48.719707 25339 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
I20250901 20:02:48.720333 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 25339
I20250901 20:02:48.720464 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:02:48.720705 14605 external_mini_cluster.cc:1468] Setting key 4bf8ac1b4d7972d2f6f67beda99e16c7
I20250901 20:02:48.725347 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:48.721551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:35535 (local address 127.14.67.126:39283)
0901 20:02:48.721677 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:48.721681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:48.722532 (+ 851us) server_negotiation.cc:407] Connection header received
0901 20:02:48.723456 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:48.723459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:48.723494 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:48.723553 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:48.724760 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:48.724765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:48.724767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0901 20:02:48.724782 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:48.724811 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0901 20:02:48.724820 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:48.724946 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:48.725198 (+ 252us) server_negotiation.cc:299] Negotiation successful
0901 20:02:48.725239 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20250901 20:02:48.725976 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:48.721803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:39283 (local address 127.14.67.67:35535)
0901 20:02:48.722392 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:48.722424 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:48.723262 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:48.723716 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:48.723726 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:48.723819 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:48.724493 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0901 20:02:48.724497 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:48.724535 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0901 20:02:48.724584 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:48.724979 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:48.725039 (+ 60us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:48.725046 (+ 7us) client_negotiation.cc:769] Sending connection context
0901 20:02:48.725219 (+ 173us) client_negotiation.cc:240] Negotiation successful
0901 20:02:48.725397 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
I20250901 20:02:48.727229 25467 heartbeater.cc:344] Connected to a master server at 127.14.67.126:39283
I20250901 20:02:48.727483 25467 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:48.727923 25467 heartbeater.cc:507] Master 127.14.67.126:39283 requested a full tablet report, sending...
I20250901 20:02:48.729086 25017 ts_manager.cc:194] Registered new tserver with Master: 31b835f05b33430d893101426c80ab11 (127.14.67.67:36629)
I20250901 20:02:48.736097 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250901 20:02:48.738576 25056 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:02:48.737793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49512 (local address 127.14.67.126:39283)
0901 20:02:48.737907 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:48.737910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:48.738056 (+ 146us) server_negotiation.cc:407] Connection header received
0901 20:02:48.738144 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:48.738146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:48.738176 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:48.738243 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:48.738453 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49512: BlockingRecv error: recv got EOF from 127.0.0.1:49512 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20250901 20:02:48.738935 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 25077
I20250901 20:02:48.744541 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 25208
I20250901 20:02:48.750007 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 25339
I20250901 20:02:48.754957 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 24986
2025-09-01T20:02:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (719 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:02:48 dist-test-slave-2s4d krb5kdc[25477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:02:48 dist-test-slave-2s4d krb5kdc[25477](info): set up 2 sockets
Sep 01 20:02:48 dist-test-slave-2s4d krb5kdc[25477](info): commencing operation
krb5kdc: starting...
W20250901 20:02:50.755951 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.980s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:02:50 dist-test-slave-2s4d krb5kdc[25477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:02:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:02:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:50.893289 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:35933
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:36099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:35933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:50.994458 25493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:50.994676 25493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:50.994755 25493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:50.997684 25493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:02:50.997736 25493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:50.997776 25493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:50.997815 25493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:02:50.997855 25493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:02:51.001722 25493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:36099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:35933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:35933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:51.002810 25493 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:51.003611 25493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:51.009023 25501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:51.009012 25498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:51.009150 25493 server_base.cc:1047] running on GCE node
W20250901 20:02:51.009012 25499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:51.009897 25493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:51.011013 25493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:51.012195 25493 hybrid_clock.cc:648] HybridClock initialized: now 1756756971012164 us; error 44 us; skew 500 ppm
Sep 01 20:02:51 dist-test-slave-2s4d krb5kdc[25477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:51.015163 25493 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:02:51.016338 25493 webserver.cc:480] Webserver started at http://127.14.67.126:39677/ using document root <none> and password file <none>
I20250901 20:02:51.016952 25493 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:51.017006 25493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:51.017207 25493 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:51.019192 25493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "a70daf65ed96467fae25d59d55474e73"
format_stamp: "Formatted at 2025-09-01 20:02:51 on dist-test-slave-2s4d"
server_key: "613737ddab6c0358a44ae149412cdfd7"
server_key_iv: "d3a54d77760e0652327488c0877fee51"
server_key_version: "encryptionkey@0"
I20250901 20:02:51.019706 25493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "a70daf65ed96467fae25d59d55474e73"
format_stamp: "Formatted at 2025-09-01 20:02:51 on dist-test-slave-2s4d"
server_key: "613737ddab6c0358a44ae149412cdfd7"
server_key_iv: "d3a54d77760e0652327488c0877fee51"
server_key_version: "encryptionkey@0"
I20250901 20:02:51.023494 25493 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250901 20:02:51.025588 25508 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:51.026465 25493 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250901 20:02:51.026616 25493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "a70daf65ed96467fae25d59d55474e73"
format_stamp: "Formatted at 2025-09-01 20:02:51 on dist-test-slave-2s4d"
server_key: "613737ddab6c0358a44ae149412cdfd7"
server_key_iv: "d3a54d77760e0652327488c0877fee51"
server_key_version: "encryptionkey@0"
I20250901 20:02:51.026721 25493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:51.037997 25493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:51.044274 25493 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:51.044442 25493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:51.052336 25493 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:35933
I20250901 20:02:51.052316 25560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:35933 every 8 connection(s)
I20250901 20:02:51.053268 25493 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:02:51.056543 25561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:51.058226 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 25493
I20250901 20:02:51.058377 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:02:51.058729 14605 external_mini_cluster.cc:1468] Setting key 4b1d1df7814629728e60cb636b06f5fd
I20250901 20:02:51.063465 25561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73: Bootstrap starting.
I20250901 20:02:51.065650 25561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73: Neither blocks nor log segments found. Creating new log.
Sep 01 20:02:51 dist-test-slave-2s4d krb5kdc[25477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:02:51.066304 25561 log.cc:826] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:51.068392 25561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73: No bootstrap required, opened a new log
I20250901 20:02:51.071252 25561 raft_consensus.cc:357] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a70daf65ed96467fae25d59d55474e73" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35933 } }
I20250901 20:02:51.071455 25561 raft_consensus.cc:383] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:51.071516 25561 raft_consensus.cc:738] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a70daf65ed96467fae25d59d55474e73, State: Initialized, Role: FOLLOWER
I20250901 20:02:51.072016 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.060170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.14.67.126:35933)
0901 20:02:51.060672 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:51.060681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:51.060716 (+ 35us) server_negotiation.cc:407] Connection header received
0901 20:02:51.061324 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:51.061340 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:51.061612 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:51.061905 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:51.062879 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.063670 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.064302 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.064577 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.067017 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:51.067033 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:51.067041 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:51.067072 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:51.069300 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:51.069784 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:51.069790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:51.069795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:51.069876 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:51.070141 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:51.070144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:51.070146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:51.070536 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:51.070726 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:51.071054 (+ 328us) server_negotiation.cc:299] Negotiation successful
0901 20:02:51.071291 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":159,"threads_started":1}
I20250901 20:02:51.071966 25561 consensus_queue.cc:260] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a70daf65ed96467fae25d59d55474e73" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35933 } }
I20250901 20:02:51.072124 25561 raft_consensus.cc:397] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:02:51.072180 25561 raft_consensus.cc:491] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:02:51.072914 25561 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:51.073693 25561 raft_consensus.cc:513] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a70daf65ed96467fae25d59d55474e73" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35933 } }
I20250901 20:02:51.074010 25561 leader_election.cc:304] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a70daf65ed96467fae25d59d55474e73; no voters:
I20250901 20:02:51.074301 25561 leader_election.cc:290] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:02:51.074414 25566 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:51.074651 25566 raft_consensus.cc:695] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [term 1 LEADER]: Becoming Leader. State: Replica: a70daf65ed96467fae25d59d55474e73, State: Running, Role: LEADER
I20250901 20:02:51.074957 25566 consensus_queue.cc:237] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a70daf65ed96467fae25d59d55474e73" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35933 } }
I20250901 20:02:51.075244 25561 sys_catalog.cc:564] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:02:51.076349 25568 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a70daf65ed96467fae25d59d55474e73. Latest consensus state: current_term: 1 leader_uuid: "a70daf65ed96467fae25d59d55474e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a70daf65ed96467fae25d59d55474e73" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35933 } } }
I20250901 20:02:51.076464 25568 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:51.077474 25567 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a70daf65ed96467fae25d59d55474e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a70daf65ed96467fae25d59d55474e73" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 35933 } } }
I20250901 20:02:51.077540 25575 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:02:51.077565 25567 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73 [sys.catalog]: This master's current role is: LEADER
I20250901 20:02:51.080302 25575 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:02:51.084914 25575 catalog_manager.cc:1349] Generated new cluster ID: 235faff370924ada913b2dbb58bcaa04
I20250901 20:02:51.084978 25575 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:02:51.104228 25575 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:02:51.105238 25575 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:02:51.112426 25575 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73: Generated new TSK 0
I20250901 20:02:51.112988 25575 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:51.163506 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:35933
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:36099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:51.274665 25589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:51.274860 25589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:51.274948 25589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:51.278084 25589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:51.278154 25589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:51.278264 25589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:02:51.282102 25589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:36099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:35933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:51.283094 25589 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:51.283874 25589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:51.289800 25594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:51.289791 25595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:51.289855 25597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:51.290196 25589 server_base.cc:1047] running on GCE node
I20250901 20:02:51.290706 25589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:51.291366 25589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:51.292559 25589 hybrid_clock.cc:648] HybridClock initialized: now 1756756971292496 us; error 74 us; skew 500 ppm
Sep 01 20:02:51 dist-test-slave-2s4d krb5kdc[25477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:51.295181 25589 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:02:51.296316 25589 webserver.cc:480] Webserver started at http://127.14.67.65:34921/ using document root <none> and password file <none>
I20250901 20:02:51.296828 25589 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:51.296890 25589 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:51.297056 25589 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:51.298748 25589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "c2a095c2a7b1406aa389ff6722708819"
format_stamp: "Formatted at 2025-09-01 20:02:51 on dist-test-slave-2s4d"
server_key: "bd9a729ccaebce4088ef82d6d989b110"
server_key_iv: "558ec5b7bfffe1db39434f6b814794c9"
server_key_version: "encryptionkey@0"
I20250901 20:02:51.299206 25589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "c2a095c2a7b1406aa389ff6722708819"
format_stamp: "Formatted at 2025-09-01 20:02:51 on dist-test-slave-2s4d"
server_key: "bd9a729ccaebce4088ef82d6d989b110"
server_key_iv: "558ec5b7bfffe1db39434f6b814794c9"
server_key_version: "encryptionkey@0"
I20250901 20:02:51.302235 25589 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250901 20:02:51.304435 25604 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:51.305539 25589 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:02:51.305653 25589 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "c2a095c2a7b1406aa389ff6722708819"
format_stamp: "Formatted at 2025-09-01 20:02:51 on dist-test-slave-2s4d"
server_key: "bd9a729ccaebce4088ef82d6d989b110"
server_key_iv: "558ec5b7bfffe1db39434f6b814794c9"
server_key_version: "encryptionkey@0"
I20250901 20:02:51.305737 25589 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:51.320669 25589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:51.323102 25589 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:51.323251 25589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:51.323714 25589 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:51.324455 25589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:51.324501 25589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:51.324556 25589 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:51.324584 25589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:51.332952 25589 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:36327
I20250901 20:02:51.332983 25717 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:36327 every 8 connection(s)
I20250901 20:02:51.333760 25589 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:02:51.339332 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 25589
I20250901 20:02:51.339468 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:02:51.339716 14605 external_mini_cluster.cc:1468] Setting key 97b058b6e0c1e46aa2c5a8fcf3a39b3a
Sep 01 20:02:51 dist-test-slave-2s4d krb5kdc[25477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:02:51.345729 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.335277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:58141 (local address 127.14.67.126:35933)
0901 20:02:51.335414 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:51.335419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:51.335997 (+ 578us) server_negotiation.cc:407] Connection header received
0901 20:02:51.336723 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:51.336730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:51.336776 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:51.336849 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:51.338293 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.338903 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.339498 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.339680 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.342013 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:51.342026 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:51.342028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:51.342049 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:51.343863 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:51.344290 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:51.344296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:51.344301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:51.344353 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:51.344663 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:51.344668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:51.344672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:51.344864 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:51.344966 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:51.345366 (+ 400us) server_negotiation.cc:299] Negotiation successful
0901 20:02:51.345504 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20250901 20:02:51.346065 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.335499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:35933 (local address 127.14.67.65:58141)
0901 20:02:51.335868 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:51.335900 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:51.336540 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:51.336964 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:51.336971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:51.337286 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:51.338122 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.338134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.338999 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.339001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:51.339390 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.339400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.339575 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.340141 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:51.340164 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:51.341869 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:51.343974 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:51.343978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:51.343985 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:51.344197 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:51.344448 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:51.344453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:51.344457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:51.344570 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:51.344956 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:51.344960 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:51.345142 (+ 182us) client_negotiation.cc:769] Sending connection context
0901 20:02:51.345291 (+ 149us) client_negotiation.cc:240] Negotiation successful
0901 20:02:51.345469 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":115,"threads_started":1}
I20250901 20:02:51.347040 25718 heartbeater.cc:344] Connected to a master server at 127.14.67.126:35933
I20250901 20:02:51.347285 25718 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:51.347833 25718 heartbeater.cc:507] Master 127.14.67.126:35933 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
I20250901 20:02:51.349239 25519 ts_manager.cc:194] Registered new tserver with Master: c2a095c2a7b1406aa389ff6722708819 (127.14.67.65:36327)
I20250901 20:02:51.350286 25519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:58141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:51.383673 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:35933
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:36099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:51.497537 25725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:51.497797 25725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:51.497859 25725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:51.501072 25725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:51.501142 25725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:51.501225 25725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:02:51.505599 25725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:36099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:35933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:51.506907 25725 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:51.507896 25725 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:51.515681 25730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:51.515684 25731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:51.515681 25733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:51.516116 25725 server_base.cc:1047] running on GCE node
I20250901 20:02:51.516523 25725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:51.517328 25725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:51.518591 25725 hybrid_clock.cc:648] HybridClock initialized: now 1756756971518512 us; error 54 us; skew 500 ppm
Sep 01 20:02:51 dist-test-slave-2s4d krb5kdc[25477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:51.521374 25725 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:02:51.522420 25725 webserver.cc:480] Webserver started at http://127.14.67.66:40221/ using document root <none> and password file <none>
I20250901 20:02:51.522979 25725 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:51.523027 25725 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:51.523180 25725 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:51.524853 25725 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "c82e1a06a761499db603a2893be12178"
format_stamp: "Formatted at 2025-09-01 20:02:51 on dist-test-slave-2s4d"
server_key: "eb308d127c90a54bfc4d23e695dcdb1d"
server_key_iv: "342a41eadc8eafdfbbc029f60c5c7c5e"
server_key_version: "encryptionkey@0"
I20250901 20:02:51.525290 25725 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "c82e1a06a761499db603a2893be12178"
format_stamp: "Formatted at 2025-09-01 20:02:51 on dist-test-slave-2s4d"
server_key: "eb308d127c90a54bfc4d23e695dcdb1d"
server_key_iv: "342a41eadc8eafdfbbc029f60c5c7c5e"
server_key_version: "encryptionkey@0"
I20250901 20:02:51.528597 25725 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250901 20:02:51.530786 25740 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:51.532059 25725 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250901 20:02:51.532171 25725 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "c82e1a06a761499db603a2893be12178"
format_stamp: "Formatted at 2025-09-01 20:02:51 on dist-test-slave-2s4d"
server_key: "eb308d127c90a54bfc4d23e695dcdb1d"
server_key_iv: "342a41eadc8eafdfbbc029f60c5c7c5e"
server_key_version: "encryptionkey@0"
I20250901 20:02:51.532292 25725 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:51.542191 25725 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:51.544936 25725 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:51.545089 25725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:51.545588 25725 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:51.546384 25725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:51.546434 25725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:51.546490 25725 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:51.546550 25725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:51.555384 25725 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:43473
I20250901 20:02:51.555436 25853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:43473 every 8 connection(s)
I20250901 20:02:51.556272 25725 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
I20250901 20:02:51.559603 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 25725
I20250901 20:02:51.559741 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:02:51.560014 14605 external_mini_cluster.cc:1468] Setting key c11aa73856ba8f61d66709ccbff6f137
Sep 01 20:02:51 dist-test-slave-2s4d krb5kdc[25477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.67.67@KRBTEST.COM; defaulting to no policy
I20250901 20:02:51.570199 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.558171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.66:47767 (local address 127.14.67.126:35933)
0901 20:02:51.558335 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:51.558340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:51.558931 (+ 591us) server_negotiation.cc:407] Connection header received
0901 20:02:51.559971 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:51.559975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:51.560054 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:51.560193 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:51.562011 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.562494 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.563133 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.563308 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.566182 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:51.566207 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:51.566210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:51.566238 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:51.568166 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:51.568690 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:51.568692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:51.568694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:51.568737 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:51.569178 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:51.569181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:51.569183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:51.569354 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:51.569457 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:51.569985 (+ 528us) server_negotiation.cc:299] Negotiation successful
0901 20:02:51.570089 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20250901 20:02:51.570827 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.558330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:35933 (local address 127.14.67.66:47767)
0901 20:02:51.558797 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:51.558832 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:51.559608 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:51.560430 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:51.560495 (+ 65us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:51.560930 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:51.561870 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.561886 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.562628 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.562632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:51.563020 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.563026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.563191 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.563935 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:51.563970 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:51.565987 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:51.568303 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:51.568309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:51.568322 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:51.568580 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:51.568879 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:51.568883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:51.568886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:51.569041 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:51.569497 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:51.569502 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:51.569741 (+ 239us) client_negotiation.cc:769] Sending connection context
0901 20:02:51.569920 (+ 179us) client_negotiation.cc:240] Negotiation successful
0901 20:02:51.570142 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.67@KRBTEST.COM" created.
I20250901 20:02:51.571986 25854 heartbeater.cc:344] Connected to a master server at 127.14.67.126:35933
I20250901 20:02:51.572206 25854 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:51.572778 25854 heartbeater.cc:507] Master 127.14.67.126:35933 requested a full tablet report, sending...
I20250901 20:02:51.573773 25519 ts_manager.cc:194] Registered new tserver with Master: c82e1a06a761499db603a2893be12178 (127.14.67.66:43473)
I20250901 20:02:51.574343 25519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.66@KRBTEST.COM'} at 127.14.67.66:47767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:02:51.607795 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.67:0
--local_ip_for_outbound_sockets=127.14.67.67
--webserver_interface=127.14.67.67
--webserver_port=0
--tserver_master_addrs=127.14.67.126:35933
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:36099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:02:51.715099 25861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:02:51.715287 25861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:02:51.715373 25861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:02:51.718345 25861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:02:51.718403 25861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:02:51.718523 25861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.67
I20250901 20:02:51.722340 25861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:36099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.67.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:35933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.67
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:02:51.723337 25861 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:02:51.724120 25861 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:02:51.731137 25867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:51.731132 25866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:02:51.731132 25869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:02:51.731642 25861 server_base.cc:1047] running on GCE node
I20250901 20:02:51.732065 25861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:02:51.732687 25861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:02:51.733870 25861 hybrid_clock.cc:648] HybridClock initialized: now 1756756971733847 us; error 43 us; skew 500 ppm
Sep 01 20:02:51 dist-test-slave-2s4d krb5kdc[25477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:02:51.736718 25861 init.cc:377] Logged in from keytab as kudu/127.14.67.67@KRBTEST.COM (short username kudu)
I20250901 20:02:51.737758 25861 webserver.cc:480] Webserver started at http://127.14.67.67:37461/ using document root <none> and password file <none>
I20250901 20:02:51.738263 25861 fs_manager.cc:362] Metadata directory not provided
I20250901 20:02:51.738324 25861 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:02:51.738541 25861 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250901 20:02:51.740185 25861 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/instance:
uuid: "e07f18c205414e87be5541c9296343fc"
format_stamp: "Formatted at 2025-09-01 20:02:51 on dist-test-slave-2s4d"
server_key: "120ff47c4fceb15deb887b2e7250ea0f"
server_key_iv: "6db20271e51cc9e206cf5acc6d208796"
server_key_version: "encryptionkey@0"
I20250901 20:02:51.740672 25861 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance:
uuid: "e07f18c205414e87be5541c9296343fc"
format_stamp: "Formatted at 2025-09-01 20:02:51 on dist-test-slave-2s4d"
server_key: "120ff47c4fceb15deb887b2e7250ea0f"
server_key_iv: "6db20271e51cc9e206cf5acc6d208796"
server_key_version: "encryptionkey@0"
I20250901 20:02:51.743957 25861 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250901 20:02:51.746161 25876 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:51.747388 25861 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250901 20:02:51.747511 25861 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
uuid: "e07f18c205414e87be5541c9296343fc"
format_stamp: "Formatted at 2025-09-01 20:02:51 on dist-test-slave-2s4d"
server_key: "120ff47c4fceb15deb887b2e7250ea0f"
server_key_iv: "6db20271e51cc9e206cf5acc6d208796"
server_key_version: "encryptionkey@0"
I20250901 20:02:51.747601 25861 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:02:51.758698 25861 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:02:51.761353 25861 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:02:51.761521 25861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:02:51.762008 25861 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:02:51.762833 25861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:02:51.762892 25861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:51.762953 25861 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:02:51.762984 25861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:02:51.773756 25861 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.67:45905
I20250901 20:02:51.773783 25989 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.67:45905 every 8 connection(s)
I20250901 20:02:51.774864 25861 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/data/info.pb
Sep 01 20:02:51 dist-test-slave-2s4d krb5kdc[25477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.67@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:02:51.783727 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 25861
I20250901 20:02:51.783851 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal/instance
I20250901 20:02:51.784152 14605 external_mini_cluster.cc:1468] Setting key 3825de5665e49b77c1a25104587ac025
I20250901 20:02:51.786653 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.776570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.67:36325 (local address 127.14.67.126:35933)
0901 20:02:51.776702 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:51.776706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:51.777329 (+ 623us) server_negotiation.cc:407] Connection header received
0901 20:02:51.778156 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:51.778160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:51.778220 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:51.778296 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:51.779832 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.780313 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.780921 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.781096 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.783080 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:51.783096 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:51.783098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:51.783120 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:51.784574 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:51.785353 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:51.785358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:51.785361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:51.785405 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:51.785681 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:51.785683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:51.785684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:51.785811 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:51.785890 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:51.786414 (+ 524us) server_negotiation.cc:299] Negotiation successful
0901 20:02:51.786501 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20250901 20:02:51.787205 25992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.776766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:35933 (local address 127.14.67.67:36325)
0901 20:02:51.777182 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:51.777215 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:51.777976 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:51.778449 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:51.778456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:51.778836 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:02:51.779692 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.779705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.780421 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.780424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:51.780812 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.780818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.780987 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.781533 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:02:51.781557 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:02:51.782940 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:02:51.784774 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:51.784784 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:51.784797 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:51.785256 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:51.785492 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:02:51.785496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:02:51.785498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:02:51.785600 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:02:51.785899 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:02:51.785905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:02:51.786162 (+ 257us) client_negotiation.cc:769] Sending connection context
0901 20:02:51.786365 (+ 203us) client_negotiation.cc:240] Negotiation successful
0901 20:02:51.786591 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":119,"threads_started":1}
I20250901 20:02:51.788527 25990 heartbeater.cc:344] Connected to a master server at 127.14.67.126:35933
I20250901 20:02:51.788739 25990 heartbeater.cc:461] Registering TS with master...
I20250901 20:02:51.789186 25990 heartbeater.cc:507] Master 127.14.67.126:35933 requested a full tablet report, sending...
I20250901 20:02:51.790125 25519 ts_manager.cc:194] Registered new tserver with Master: e07f18c205414e87be5541c9296343fc (127.14.67.67:45905)
I20250901 20:02:51.790865 25519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.67@KRBTEST.COM'} at 127.14.67.67:36325
I20250901 20:02:51.797958 14605 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250901 20:02:51.806691 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.799734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.14.67.126:35933)
0901 20:02:51.799892 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:51.799898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:51.799951 (+ 53us) server_negotiation.cc:407] Connection header received
0901 20:02:51.800087 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:51.800094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:51.800137 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:51.800209 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:02:51.801031 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.801497 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.802125 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.802329 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.803639 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:02:51.803741 (+ 102us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:02:51.803749 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:02:51.803783 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:02:51.805282 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:51.805683 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:51.805686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:51.805688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:51.805728 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:02:51.805946 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:02:51.805949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:02:51.805951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:02:51.806095 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:02:51.806213 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:51.806403 (+ 190us) server_negotiation.cc:299] Negotiation successful
0901 20:02:51.806543 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20250901 20:02:51.809556 25519 catalog_manager.cc:2232] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 20:02:51.812630 25519 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 20:02:51.824983 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.820821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:45905 (local address 127.0.0.1:55234)
0901 20:02:51.821561 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:51.821575 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:51.821661 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:51.822150 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:51.822153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:51.822166 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:51.822467 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.822475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.823546 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.823550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:51.824253 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.824261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.824367 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.824629 (+ 262us) client_negotiation.cc:769] Sending connection context
0901 20:02:51.824723 (+ 94us) client_negotiation.cc:240] Negotiation successful
0901 20:02:51.824802 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":109,"threads_started":1}
I20250901 20:02:51.825246 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.820721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:36327 (local address 127.0.0.1:40626)
0901 20:02:51.821236 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:51.821277 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:51.821408 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:51.822120 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:51.822126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:51.822146 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:51.822740 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.822752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.824051 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.824055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:51.824888 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.824896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.824990 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.825011 (+ 21us) client_negotiation.cc:769] Sending connection context
0901 20:02:51.825062 (+ 51us) client_negotiation.cc:240] Negotiation successful
0901 20:02:51.825129 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"spinlock_wait_cycles":3840,"thread_start_us":68,"threads_started":1}
I20250901 20:02:51.825781 26005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.821368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.14.67.67:45905)
0901 20:02:51.821749 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:51.821756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:51.821774 (+ 18us) server_negotiation.cc:407] Connection header received
0901 20:02:51.821845 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:51.821852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:51.822001 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:51.822323 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:51.822590 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.823449 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.824389 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.825380 (+ 991us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.825489 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:51.825553 (+ 64us) server_negotiation.cc:299] Negotiation successful
0901 20:02:51.825631 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20250901 20:02:51.825837 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.821372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:43473 (local address 127.0.0.1:41138)
0901 20:02:51.822083 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:51.822098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:51.822165 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:51.822823 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:51.822826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:51.822846 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:51.823080 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.823087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.824288 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.824292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:51.825485 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.825496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.825602 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.825616 (+ 14us) client_negotiation.cc:769] Sending connection context
0901 20:02:51.825668 (+ 52us) client_negotiation.cc:240] Negotiation successful
0901 20:02:51.825719 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":104,"threads_started":1}
I20250901 20:02:51.826962 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.821666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.14.67.66:43473)
0901 20:02:51.822414 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:51.822420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:51.822440 (+ 20us) server_negotiation.cc:407] Connection header received
0901 20:02:51.822539 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:51.822546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:51.822692 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:51.822833 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:51.823203 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.824189 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.825713 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.826621 (+ 908us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.826711 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:51.826774 (+ 63us) server_negotiation.cc:299] Negotiation successful
0901 20:02:51.826851 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":107,"threads_started":1}
I20250901 20:02:51.827530 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.820845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40626 (local address 127.14.67.65:36327)
0901 20:02:51.821135 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:51.821141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:51.821532 (+ 391us) server_negotiation.cc:407] Connection header received
0901 20:02:51.821618 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:51.821623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:51.821802 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:51.821942 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:51.822874 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.823923 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.825593 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.827026 (+ 1433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.827160 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:51.827241 (+ 81us) server_negotiation.cc:299] Negotiation successful
0901 20:02:51.827349 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":109,"threads_started":1}
I20250901 20:02:51.829752 25788 tablet_service.cc:1468] Processing CreateTablet for tablet 2d70217fcd68421686b4db779495d29c (DEFAULT_TABLE table=test-table [id=70a9a7a731c64a2eb2dee44935637dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:51.829749 25924 tablet_service.cc:1468] Processing CreateTablet for tablet 2d70217fcd68421686b4db779495d29c (DEFAULT_TABLE table=test-table [id=70a9a7a731c64a2eb2dee44935637dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:51.830184 25652 tablet_service.cc:1468] Processing CreateTablet for tablet 2d70217fcd68421686b4db779495d29c (DEFAULT_TABLE table=test-table [id=70a9a7a731c64a2eb2dee44935637dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 20:02:51.830967 25788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d70217fcd68421686b4db779495d29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:51.830964 25924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d70217fcd68421686b4db779495d29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:51.831009 25652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d70217fcd68421686b4db779495d29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:02:51.837286 26007 tablet_bootstrap.cc:492] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc: Bootstrap starting.
I20250901 20:02:51.839566 26009 tablet_bootstrap.cc:492] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819: Bootstrap starting.
I20250901 20:02:51.840276 26007 tablet_bootstrap.cc:654] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:51.840978 26007 log.cc:826] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:51.841487 26009 tablet_bootstrap.cc:654] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:51.841662 26008 tablet_bootstrap.cc:492] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178: Bootstrap starting.
I20250901 20:02:51.842269 26009 log.cc:826] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:51.842981 26007 tablet_bootstrap.cc:492] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc: No bootstrap required, opened a new log
I20250901 20:02:51.843178 26007 ts_tablet_manager.cc:1397] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250901 20:02:51.843410 26008 tablet_bootstrap.cc:654] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178: Neither blocks nor log segments found. Creating new log.
I20250901 20:02:51.843989 26008 log.cc:826] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178: Log is configured to *not* fsync() on all Append() calls
I20250901 20:02:51.844130 26009 tablet_bootstrap.cc:492] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819: No bootstrap required, opened a new log
I20250901 20:02:51.844302 26009 ts_tablet_manager.cc:1397] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250901 20:02:51.845490 26008 tablet_bootstrap.cc:492] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178: No bootstrap required, opened a new log
I20250901 20:02:51.845642 26008 ts_tablet_manager.cc:1397] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250901 20:02:51.846853 26007 raft_consensus.cc:357] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a095c2a7b1406aa389ff6722708819" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 36327 } } peers { permanent_uuid: "e07f18c205414e87be5541c9296343fc" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 45905 } } peers { permanent_uuid: "c82e1a06a761499db603a2893be12178" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 43473 } }
I20250901 20:02:51.847097 26007 raft_consensus.cc:383] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:51.847163 26007 raft_consensus.cc:738] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07f18c205414e87be5541c9296343fc, State: Initialized, Role: FOLLOWER
I20250901 20:02:51.847270 26009 raft_consensus.cc:357] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a095c2a7b1406aa389ff6722708819" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 36327 } } peers { permanent_uuid: "e07f18c205414e87be5541c9296343fc" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 45905 } } peers { permanent_uuid: "c82e1a06a761499db603a2893be12178" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 43473 } }
I20250901 20:02:51.847451 26009 raft_consensus.cc:383] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:51.847509 26009 raft_consensus.cc:738] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a095c2a7b1406aa389ff6722708819, State: Initialized, Role: FOLLOWER
I20250901 20:02:51.847620 26007 consensus_queue.cc:260] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a095c2a7b1406aa389ff6722708819" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 36327 } } peers { permanent_uuid: "e07f18c205414e87be5541c9296343fc" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 45905 } } peers { permanent_uuid: "c82e1a06a761499db603a2893be12178" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 43473 } }
I20250901 20:02:51.847887 26009 consensus_queue.cc:260] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a095c2a7b1406aa389ff6722708819" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 36327 } } peers { permanent_uuid: "e07f18c205414e87be5541c9296343fc" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 45905 } } peers { permanent_uuid: "c82e1a06a761499db603a2893be12178" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 43473 } }
I20250901 20:02:51.848330 25990 heartbeater.cc:499] Master 127.14.67.126:35933 was elected leader, sending a full tablet report...
I20250901 20:02:51.848534 26009 ts_tablet_manager.cc:1428] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250901 20:02:51.848527 26007 ts_tablet_manager.cc:1428] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250901 20:02:51.848536 26008 raft_consensus.cc:357] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a095c2a7b1406aa389ff6722708819" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 36327 } } peers { permanent_uuid: "e07f18c205414e87be5541c9296343fc" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 45905 } } peers { permanent_uuid: "c82e1a06a761499db603a2893be12178" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 43473 } }
I20250901 20:02:51.848842 25718 heartbeater.cc:499] Master 127.14.67.126:35933 was elected leader, sending a full tablet report...
I20250901 20:02:51.848721 26008 raft_consensus.cc:383] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:02:51.849604 26008 raft_consensus.cc:738] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82e1a06a761499db603a2893be12178, State: Initialized, Role: FOLLOWER
I20250901 20:02:51.849985 26008 consensus_queue.cc:260] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a095c2a7b1406aa389ff6722708819" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 36327 } } peers { permanent_uuid: "e07f18c205414e87be5541c9296343fc" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 45905 } } peers { permanent_uuid: "c82e1a06a761499db603a2893be12178" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 43473 } }
I20250901 20:02:51.850765 25854 heartbeater.cc:499] Master 127.14.67.126:35933 was elected leader, sending a full tablet report...
I20250901 20:02:51.850818 26008 ts_tablet_manager.cc:1428] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250901 20:02:51.915041 26014 raft_consensus.cc:491] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 20:02:51.915212 26014 raft_consensus.cc:513] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a095c2a7b1406aa389ff6722708819" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 36327 } } peers { permanent_uuid: "e07f18c205414e87be5541c9296343fc" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 45905 } } peers { permanent_uuid: "c82e1a06a761499db603a2893be12178" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 43473 } }
I20250901 20:02:51.916247 26014 leader_election.cc:290] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07f18c205414e87be5541c9296343fc (127.14.67.67:45905), c82e1a06a761499db603a2893be12178 (127.14.67.66:43473)
I20250901 20:02:51.919569 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.916491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.66:43473 (local address 127.14.67.65:44401)
0901 20:02:51.916816 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:51.916829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:51.916961 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:51.917219 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:51.917222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:51.917242 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:51.917507 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.917514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.918257 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.918261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:51.919109 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.919118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.919230 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.919256 (+ 26us) client_negotiation.cc:769] Sending connection context
0901 20:02:51.919316 (+ 60us) client_negotiation.cc:240] Negotiation successful
0901 20:02:51.919377 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"spinlock_wait_cycles":2816,"thread_start_us":148,"threads_started":1}
I20250901 20:02:51.919589 26016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.916494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.67:45905 (local address 127.14.67.65:43857)
0901 20:02:51.916915 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:51.916926 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0901 20:02:51.917018 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:02:51.917229 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:02:51.917232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:02:51.917292 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0901 20:02:51.917718 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.917724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.918449 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.918452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:02:51.919194 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:02:51.919202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.919303 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.919320 (+ 17us) client_negotiation.cc:769] Sending connection context
0901 20:02:51.919370 (+ 50us) client_negotiation.cc:240] Negotiation successful
0901 20:02:51.919421 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":10,"spinlock_wait_cycles":1792,"thread_start_us":137,"threads_started":1}
I20250901 20:02:51.920023 26005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.916608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:43857 (local address 127.14.67.67:45905)
0901 20:02:51.916836 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:51.916841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:51.916988 (+ 147us) server_negotiation.cc:407] Connection header received
0901 20:02:51.917106 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:51.917110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:51.917151 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:51.917214 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:51.917823 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.918343 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.919294 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.919767 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.919807 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:51.919856 (+ 49us) server_negotiation.cc:299] Negotiation successful
0901 20:02:51.919913 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20250901 20:02:51.920173 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.916575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:44401 (local address 127.14.67.66:43473)
0901 20:02:51.916738 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:51.916745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:51.916926 (+ 181us) server_negotiation.cc:407] Connection header received
0901 20:02:51.917079 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:51.917082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:51.917127 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:51.917200 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0901 20:02:51.917637 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.918139 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.919217 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.919724 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.919771 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:51.919829 (+ 58us) server_negotiation.cc:299] Negotiation successful
0901 20:02:51.919900 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20250901 20:02:51.920518 25944 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d70217fcd68421686b4db779495d29c" candidate_uuid: "c2a095c2a7b1406aa389ff6722708819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07f18c205414e87be5541c9296343fc" is_pre_election: true
I20250901 20:02:51.920660 25808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d70217fcd68421686b4db779495d29c" candidate_uuid: "c2a095c2a7b1406aa389ff6722708819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82e1a06a761499db603a2893be12178" is_pre_election: true
I20250901 20:02:51.920742 25944 raft_consensus.cc:2466] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a095c2a7b1406aa389ff6722708819 in term 0.
I20250901 20:02:51.920913 25808 raft_consensus.cc:2466] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a095c2a7b1406aa389ff6722708819 in term 0.
I20250901 20:02:51.921218 25606 leader_election.cc:304] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a095c2a7b1406aa389ff6722708819, e07f18c205414e87be5541c9296343fc; no voters:
I20250901 20:02:51.921442 26014 raft_consensus.cc:2802] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 20:02:51.921510 26014 raft_consensus.cc:491] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 20:02:51.921545 26014 raft_consensus.cc:3058] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:51.922439 26014 raft_consensus.cc:513] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a095c2a7b1406aa389ff6722708819" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 36327 } } peers { permanent_uuid: "e07f18c205414e87be5541c9296343fc" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 45905 } } peers { permanent_uuid: "c82e1a06a761499db603a2893be12178" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 43473 } }
I20250901 20:02:51.922801 26014 leader_election.cc:290] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [CANDIDATE]: Term 1 election: Requested vote from peers e07f18c205414e87be5541c9296343fc (127.14.67.67:45905), c82e1a06a761499db603a2893be12178 (127.14.67.66:43473)
I20250901 20:02:51.923125 25944 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d70217fcd68421686b4db779495d29c" candidate_uuid: "c2a095c2a7b1406aa389ff6722708819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07f18c205414e87be5541c9296343fc"
I20250901 20:02:51.923219 25944 raft_consensus.cc:3058] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:51.924551 25944 raft_consensus.cc:2466] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a095c2a7b1406aa389ff6722708819 in term 1.
I20250901 20:02:51.924976 25808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d70217fcd68421686b4db779495d29c" candidate_uuid: "c2a095c2a7b1406aa389ff6722708819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82e1a06a761499db603a2893be12178"
I20250901 20:02:51.925064 25606 leader_election.cc:304] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2a095c2a7b1406aa389ff6722708819, e07f18c205414e87be5541c9296343fc; no voters:
I20250901 20:02:51.925144 25808 raft_consensus.cc:3058] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:02:51.925274 26014 raft_consensus.cc:2802] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:02:51.925513 26014 raft_consensus.cc:695] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [term 1 LEADER]: Becoming Leader. State: Replica: c2a095c2a7b1406aa389ff6722708819, State: Running, Role: LEADER
I20250901 20:02:51.925812 26014 consensus_queue.cc:237] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a095c2a7b1406aa389ff6722708819" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 36327 } } peers { permanent_uuid: "e07f18c205414e87be5541c9296343fc" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 45905 } } peers { permanent_uuid: "c82e1a06a761499db603a2893be12178" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 43473 } }
I20250901 20:02:51.926362 25808 raft_consensus.cc:2466] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a095c2a7b1406aa389ff6722708819 in term 1.
I20250901 20:02:51.929240 25519 catalog_manager.cc:5582] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a095c2a7b1406aa389ff6722708819 (127.14.67.65). New cstate: current_term: 1 leader_uuid: "c2a095c2a7b1406aa389ff6722708819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a095c2a7b1406aa389ff6722708819" member_type: VOTER last_known_addr { host: "127.14.67.65" port: 36327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07f18c205414e87be5541c9296343fc" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 45905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82e1a06a761499db603a2893be12178" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 43473 } health_report { overall_health: UNKNOWN } } }
I20250901 20:02:51.949370 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.946823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40634 (local address 127.14.67.65:36327)
0901 20:02:51.946954 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0901 20:02:51.946957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0901 20:02:51.946975 (+ 18us) server_negotiation.cc:407] Connection header received
0901 20:02:51.947043 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:02:51.947046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:02:51.947101 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:02:51.947185 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0901 20:02:51.947439 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.947894 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:02:51.948502 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:02:51.948645 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:02:51.948725 (+ 80us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0901 20:02:51.949101 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0901 20:02:51.949209 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0901 20:02:51.949234 (+ 25us) server_negotiation.cc:299] Negotiation successful
0901 20:02:51.949275 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20250901 20:02:51.954334 25944 raft_consensus.cc:1273] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc [term 1 FOLLOWER]: Refusing update from remote peer c2a095c2a7b1406aa389ff6722708819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:51.954332 25808 raft_consensus.cc:1273] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178 [term 1 FOLLOWER]: Refusing update from remote peer c2a095c2a7b1406aa389ff6722708819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 20:02:51.954908 26014 consensus_queue.cc:1035] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07f18c205414e87be5541c9296343fc" member_type: VOTER last_known_addr { host: "127.14.67.67" port: 45905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:51.955077 26018 consensus_queue.cc:1035] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82e1a06a761499db603a2893be12178" member_type: VOTER last_known_addr { host: "127.14.67.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 20:02:51.961733 26023 mvcc.cc:204] Tried to move back new op lower bound from 7195676557118730240 to 7195676557010489344. Current Snapshot: MvccSnapshot[applied={T|T < 7195676557118730240}]
I20250901 20:02:51.964779 26022 mvcc.cc:204] Tried to move back new op lower bound from 7195676557118730240 to 7195676557010489344. Current Snapshot: MvccSnapshot[applied={T|T < 7195676557118730240 or (T in {7195676557118730240})}]
I20250901 20:02:51.968562 25524 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58874:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:51.968678 25524 catalog_manager.cc:2730] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58874:
table { table_name: "test-table" } modify_external_catalogs: true
I20250901 20:02:51.970790 25524 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a70daf65ed96467fae25d59d55474e73: Sending DeleteTablet for 3 replicas of tablet 2d70217fcd68421686b4db779495d29c
I20250901 20:02:51.971383 25652 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d70217fcd68421686b4db779495d29c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:51 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:40626
I20250901 20:02:51.971428 25788 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d70217fcd68421686b4db779495d29c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:51 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:41138
I20250901 20:02:51.971841 26032 tablet_replica.cc:331] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178: stopping tablet replica
I20250901 20:02:51.971824 25924 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d70217fcd68421686b4db779495d29c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 20:02:51 UTC) from {username='kudu', principal='kudu/127.14.67.126@KRBTEST.COM'} at 127.0.0.1:55234
I20250901 20:02:51.972074 26031 tablet_replica.cc:331] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819: stopping tablet replica
I20250901 20:02:51.972259 26033 tablet_replica.cc:331] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc: stopping tablet replica
I20250901 20:02:51.972484 26033 raft_consensus.cc:2241] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:51.972572 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 25589
I20250901 20:02:51.972700 26031 raft_consensus.cc:2241] T 2d70217fcd68421686b4db779495d29c P c2a095c2a7b1406aa389ff6722708819 [term 1 LEADER]: Raft consensus shutting down.
I20250901 20:02:51.972767 26033 raft_consensus.cc:2270] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:51.972502 26032 raft_consensus.cc:2241] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 20:02:51.973085 26032 raft_consensus.cc:2270] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 20:02:51.974618 26033 ts_tablet_manager.cc:1905] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:51.974790 26032 ts_tablet_manager.cc:1905] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 20:02:51.976686 26033 ts_tablet_manager.cc:1918] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:51.976764 26033 log.cc:1199] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-2/wal/wals/2d70217fcd68421686b4db779495d29c
I20250901 20:02:51.977095 26033 ts_tablet_manager.cc:1939] T 2d70217fcd68421686b4db779495d29c P e07f18c205414e87be5541c9296343fc: Deleting consensus metadata
I20250901 20:02:51.977926 26032 ts_tablet_manager.cc:1918] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 20:02:51.978016 26032 log.cc:1199] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178: Deleting WAL directory at /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1756756873705473-14605-0/minicluster-data/ts-1/wal/wals/2d70217fcd68421686b4db779495d29c
I20250901 20:02:51.977982 25510 catalog_manager.cc:4928] TS e07f18c205414e87be5541c9296343fc (127.14.67.67:45905): tablet 2d70217fcd68421686b4db779495d29c (table test-table [id=70a9a7a731c64a2eb2dee44935637dd3]) successfully deleted
I20250901 20:02:51.978354 26032 ts_tablet_manager.cc:1939] T 2d70217fcd68421686b4db779495d29c P c82e1a06a761499db603a2893be12178: Deleting consensus metadata
I20250901 20:02:51.979566 25511 catalog_manager.cc:4928] TS c82e1a06a761499db603a2893be12178 (127.14.67.66:43473): tablet 2d70217fcd68421686b4db779495d29c (table test-table [id=70a9a7a731c64a2eb2dee44935637dd3]) successfully deleted
W20250901 20:02:51.980633 25509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.67.65:36327 (error 108)
I20250901 20:02:51.980751 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 25725
I20250901 20:02:51.982865 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:02:51.982624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.65:36327 (local address 127.0.0.1:40644)
0901 20:02:51.982746 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0901 20:02:51.982797 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.65:36327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20250901 20:02:51.982988 25509 catalog_manager.cc:4671] TS c2a095c2a7b1406aa389ff6722708819 (127.14.67.65:36327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d70217fcd68421686b4db779495d29c: Network error: Client connection negotiation failed: client connection to 127.14.67.65:36327: connect: Connection refused (error 111)
I20250901 20:02:51.986155 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 25861
I20250901 20:02:51.991287 14605 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task6kzUjy/build/debug/bin/kudu with pid 25493
2025-09-01T20:02:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3237 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 01 20:02:52 dist-test-slave-2s4d krb5kdc[26037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 01 20:02:52 dist-test-slave-2s4d krb5kdc[26037](info): set up 2 sockets
Sep 01 20:02:52 dist-test-slave-2s4d krb5kdc[26037](info): commencing operation
krb5kdc: starting...
W20250901 20:02:53.988160 14605 mini_kdc.cc:121] Time spent starting KDC: real 1.975s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/test-user.keytab.
Sep 01 20:02:54 dist-test-slave-2s4d krb5kdc[26037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756756974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-09-01T20:02:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-01T20:02:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.67.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.67.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.67.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/rangeradmin_127.14.67.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.67.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/rangeradmin_127.14.67.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.67.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.67.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.67.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/rangerlookup_127.14.67.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.67.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/rangerlookup_127.14.67.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.67.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/HTTP_127.14.67.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.67.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/HTTP_127.14.67.84@KRBTEST.COM.keytab.
I20250901 20:02:54.154129 14605 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6kzUjy/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/postgres -l logfile start
2025-09-01 20:02:56.718 UTC [26068] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-09-01 20:02:56.718 UTC [26068] LOG: listening on IPv4 address "127.14.67.84", port 52563
2025-09-01 20:02:56.722 UTC [26068] LOG: listening on Unix socket "/tmp/.s.PGSQL.52563"
2025-09-01 20:02:56.726 UTC [26073] LOG: database system was shut down at 2025-09-01 20:02:55 UTC
2025-09-01 20:02:56.730 UTC [26068] LOG: database system is ready to accept connections
I20250901 20:02:58.647809 14605 mini_postgres.cc:96] Postgres bound to 52563
2025-09-01 20:02:58.654 UTC [26079] FATAL: database "slave" does not exist
127.14.67.84:52563 - accepting connections
I20250901 20:02:58.655905 14605 mini_ranger.cc:161] Starting Ranger...
I20250901 20:02:58.675583 14605 mini_ranger.cc:84] Created miniranger Postgres user
I20250901 20:02:58.747009 14605 mini_ranger.cc:87] Created ranger Postgres database
I20250901 20:02:58.747108 14605 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-admin
2025-09-01 20:02:59,043 [I] DB FLAVOR :POSTGRES
2025-09-01 20:02:59,044 [I] --------- Verifying Ranger DB connection ---------
2025-09-01 20:02:59,044 [I] Checking connection..
2025-09-01 20:02:59,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-09-01 20:02:59,383 [I] Checking connection passed.
2025-09-01 20:02:59,384 [I] --------- Verifying version history table ---------
2025-09-01 20:02:59,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-01 20:02:59,709 [I] Table x_db_version_h does not exist in database ranger
2025-09-01 20:02:59,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-01 20:03:00,023 [I] Table x_db_version_h does not exist in database ranger
2025-09-01 20:03:00,023 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-09-01 20:03:00,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-09-01 20:03:00.344 UTC [26146] WARNING: there is no transaction in progress
2025-09-01 20:03:00,351 [I] create_dbversion_catalog.sql file imported successfully
2025-09-01 20:03:00,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-09-01 20:03:00,669 [I] Table x_db_version_h already exists in database 'ranger'
2025-09-01 20:03:00,670 [I] --------- Importing Ranger Core DB Schema ---------
2025-09-01 20:03:00,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-09-01 20:03:00,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-09-01 20:03:01,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2s4d.c.gcp-upstream.internal','N') ;"
2025-09-01 20:03:01,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-09-01 20:03:01,937 [I] Table x_portal_user does not exist in database ranger
2025-09-01 20:03:01,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-09-01 20:03:02,262 [I] Table x_policy_ref_group does not exist in database ranger
2025-09-01 20:03:02,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-09-01 20:03:02,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-09-01 20:03:02,913 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-09-01 20:03:02,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-09-01 20:03:03.776 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.788 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.799 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.809 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.825 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.836 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.882 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.887 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.897 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.906 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.915 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.924 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.932 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.937 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.941 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:03.945 UTC [26263] WARNING: there is no transaction in progress
2025-09-01 20:03:04,384 [I] ranger_core_db_postgres.sql file imported successfully
2025-09-01 20:03:04,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-09-01 20:03:04,723 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-09-01 20:03:04,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-09-01 20:03:05,052 [I] Table x_portal_user already exists in database 'ranger'
2025-09-01 20:03:05,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-09-01 20:03:05,391 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-09-01 20:03:05,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-09-01 20:03:05,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-09-01 20:03:06,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2s4d.c.gcp-upstream.internal';"
2025-09-01 20:03:06,371 [I] CORE_DB_SCHEMA import status has been updated
2025-09-01 20:03:06,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-09-01 20:03:06,720 [I] DB_PATCHES have already been applied
I20250901 20:03:06.725732 14605 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-admin:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6kzUjy/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20250901 20:03:06.725800 14605 mini_ranger.cc:193] Using host: 127.14.67.84
I20250901 20:03:06.728883 14605 mini_ranger.cc:227] Ranger admin URL: http://127.14.67.84:57481
Sep 01, 2025 8:03:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 01, 2025 8:03:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 01, 2025 8:03:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-admin/ews/webapp
Sep 01, 2025 8:03:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 01, 2025 8:03:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 01, 2025 8:03:07 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Sep 01, 2025 8:03:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-admin/ews/webapp].
Sep 01, 2025 8:03:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Sep 01, 2025 8:03:07 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-57481"]
Sep 01, 2025 8:03:07 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 01, 2025 8:03:07 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Sep 01, 2025 8:03:07 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Sep 01, 2025 8:03:07 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Sep 01, 2025 8:03:08 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Sep 01, 2025 8:03:08 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Sep 01, 2025 8:03:08 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Sep 01, 2025 8:03:08 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
I20250901 20:03:08.915786 14605 mini_ranger.cc:160] Time spent starting Ranger: real 10.260s user 0.003s sys 0.007s
Sep 01, 2025 8:03:08 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Sep 01, 2025 8:03:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 01, 2025 8:03:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
Sep 01, 2025 8:03:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 01, 2025 8:03:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 01, 2025 8:03:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 01, 2025 8:03:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Sep 01, 2025 8:03:28 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-57481"]
I20250901 20:03:28.772854 14605 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.67.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.67.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.67.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/rangerkms_127.14.67.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.67.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/rangerkms_127.14.67.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.67.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.67.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/HTTP_127.14.67.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.67.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/HTTP_127.14.67.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 01 20:03:28 dist-test-slave-2s4d krb5kdc[26037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757008, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20250901 20:03:28.860862 14605 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20250901 20:03:28.869390 14605 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20250901 20:03:28.975250 14605 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20250901 20:03:28.975344 14605 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-kms
I20250901 20:03:28.975397 14605 mini_ranger_kms.cc:227] Using postgres at 127.14.67.84:52563
2025-09-01 20:03:29,161 [I] DB FLAVOR :POSTGRES
2025-09-01 20:03:29,161 [I] --------- Verifying Ranger DB connection ---------
2025-09-01 20:03:29,161 [I] Checking connection
2025-09-01 20:03:29,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-09-01 20:03:29,474 [I] connection success
2025-09-01 20:03:29,474 [I] --------- Verifying Ranger DB tables ---------
2025-09-01 20:03:29,474 [I] Verifying table ranger_masterkey in database rangerkms
2025-09-01 20:03:29,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-09-01 20:03:29,781 [I] Table ranger_masterkey does not exist in database rangerkms
2025-09-01 20:03:29,781 [I] --------- Importing Ranger Core DB Schema ---------
2025-09-01 20:03:29,781 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-09-01 20:03:29,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.67.84:52563/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-09-01 20:03:30,073 [I] kms_core_db_postgres.sql DB schema imported successfully
I20250901 20:03:30.528689 14605 mini_ranger_kms.cc:310] Created kms service
I20250901 20:03:30.747217 14605 mini_ranger_kms.cc:326] Created kudu user
I20250901 20:03:30.787855 14605 mini_ranger_kms.cc:343] Created rangerkms user
Sep 01 20:03:30 dist-test-slave-2s4d krb5kdc[26037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757008, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.67.84@KRBTEST.COM
I20250901 20:03:31.126585 14605 mini_ranger_kms.cc:384] Added ranger policy
I20250901 20:03:31.126762 14605 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-kms:/tmp/dist-test-task6kzUjy/build/debug/bin/postgresql.jar:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6kzUjy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20250901 20:03:31.126789 14605 mini_ranger_kms.cc:240] Using host: 127.14.67.84
I20250901 20:03:31.130223 14605 mini_ranger_kms.cc:276] Ranger KMS PID: 26455
I20250901 20:03:31.130301 14605 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.67.84:36277
Sep 01, 2025 8:03:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 01, 2025 8:03:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 01, 2025 8:03:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 01, 2025 8:03:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 01, 2025 8:03:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-kms/ews/webapp].
Sep 01, 2025 8:03:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 01, 2025 8:03:32 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-36277"]
Sep 01, 2025 8:03:32 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 01, 2025 8:03:32 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Sep 01, 2025 8:03:33 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20250901 20:03:33.339377 14605 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.479s user 0.003s sys 0.166s
I20250901 20:03:33.339504 14605 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20250901 20:03:33.339597 14605 mini_ranger_kms.cc:401] 127.14.67.84:36277/kms/v1/keys
Sep 01, 2025 8:03:33 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Sep 01 20:03:33 dist-test-slave-2s4d krb5kdc[26037](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1756757013, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.67.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 01 20:03:33 dist-test-slave-2s4d krb5kdc[26037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757013, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.67.84@KRBTEST.COM for HTTP/127.14.67.84@KRBTEST.COM
Sep 01 20:03:34 dist-test-slave-2s4d krb5kdc[26037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757013, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.67.84@KRBTEST.COM for HTTP/127.14.67.84@KRBTEST.COM
Sep 01 20:03:36 dist-test-slave-2s4d krb5kdc[26037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:03:36.471284 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:46107
--webserver_interface=127.14.67.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:38159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.67.126:46107
--ranger_config_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.67.84:36277/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:03:36.597316 26508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:03:36.597513 26508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:03:36.597563 26508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:03:36.600709 26508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 20:03:36.600765 26508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:03:36.600787 26508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:03:36.600805 26508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 20:03:36.600828 26508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 20:03:36.604984 26508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:38159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.67.84:36277/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.67.126:46107
--ranger_config_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.67.126:46107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.67.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:03:36.605968 26508 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:03:36.606797 26508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:03:36.612223 26513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:03:36.612241 26514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:03:36.612238 26516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:03:36.612375 26508 server_base.cc:1047] running on GCE node
I20250901 20:03:36.612950 26508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:03:36.613785 26508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:03:36.614936 26508 hybrid_clock.cc:648] HybridClock initialized: now 1756757016614926 us; error 31 us; skew 500 ppm
Sep 01 20:03:36 dist-test-slave-2s4d krb5kdc[26037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757016, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:03:36.617455 26508 init.cc:377] Logged in from keytab as kudu/127.14.67.126@KRBTEST.COM (short username kudu)
I20250901 20:03:36.618332 26508 webserver.cc:480] Webserver started at http://127.14.67.126:39475/ using document root <none> and password file <none>
I20250901 20:03:36.618854 26508 fs_manager.cc:362] Metadata directory not provided
I20250901 20:03:36.618909 26508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:03:36.619079 26508 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 01 20:03:36 dist-test-slave-2s4d krb5kdc[26037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757016, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.126@KRBTEST.COM for HTTP/127.14.67.84@KRBTEST.COM
I20250901 20:03:36.660802 26508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/data/instance:
uuid: "0d5c35c3ac9e4692a519eb6b7a543e74"
format_stamp: "Formatted at 2025-09-01 20:03:36 on dist-test-slave-2s4d"
server_key: "3777400865c35f7a5f78410d27663cca"
server_key_iv: "0b1a9091b73dd5a7677f5d9830954497"
server_key_version: "kuduclusterkey@0"
I20250901 20:03:36.661298 26508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/wal/instance:
uuid: "0d5c35c3ac9e4692a519eb6b7a543e74"
format_stamp: "Formatted at 2025-09-01 20:03:36 on dist-test-slave-2s4d"
server_key: "3777400865c35f7a5f78410d27663cca"
server_key_iv: "0b1a9091b73dd5a7677f5d9830954497"
server_key_version: "kuduclusterkey@0"
I20250901 20:03:36.664645 26508 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250901 20:03:36.680047 26523 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:03:36.681067 26508 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250901 20:03:36.681169 26508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/wal
uuid: "0d5c35c3ac9e4692a519eb6b7a543e74"
format_stamp: "Formatted at 2025-09-01 20:03:36 on dist-test-slave-2s4d"
server_key: "3777400865c35f7a5f78410d27663cca"
server_key_iv: "0b1a9091b73dd5a7677f5d9830954497"
server_key_version: "kuduclusterkey@0"
I20250901 20:03:36.681274 26508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:03:36.719590 26508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:03:36.722888 26508 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:03:36.723066 26508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:03:36.731370 26508 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.126:46107
I20250901 20:03:36.731402 26575 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.126:46107 every 8 connection(s)
I20250901 20:03:36.732343 26508 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/data/info.pb
I20250901 20:03:36.734766 26576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 20:03:36.737946 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 26508
I20250901 20:03:36.738095 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/wal/instance
I20250901 20:03:36.738399 14605 external_mini_cluster.cc:1468] Setting key 1d5d6a224fe9755075526b270d4c16e0
I20250901 20:03:36.739691 26576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74: Bootstrap starting.
I20250901 20:03:36.741806 26576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74: Neither blocks nor log segments found. Creating new log.
I20250901 20:03:36.742748 26576 log.cc:826] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74: Log is configured to *not* fsync() on all Append() calls
I20250901 20:03:36.744421 26576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74: No bootstrap required, opened a new log
Sep 01 20:03:36 dist-test-slave-2s4d krb5kdc[26037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:03:36.747803 26576 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5c35c3ac9e4692a519eb6b7a543e74" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46107 } }
I20250901 20:03:36.748054 26576 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 20:03:36.748121 26576 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5c35c3ac9e4692a519eb6b7a543e74, State: Initialized, Role: FOLLOWER
I20250901 20:03:36.748597 26576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5c35c3ac9e4692a519eb6b7a543e74" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46107 } }
I20250901 20:03:36.748746 26576 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250901 20:03:36.748835 26576 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250901 20:03:36.748945 26576 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [term 0 FOLLOWER]: Advancing to term 1
I20250901 20:03:36.749918 26576 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5c35c3ac9e4692a519eb6b7a543e74" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46107 } }
I20250901 20:03:36.750229 26576 leader_election.cc:304] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d5c35c3ac9e4692a519eb6b7a543e74; no voters:
I20250901 20:03:36.750488 26576 leader_election.cc:290] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250901 20:03:36.750680 26581 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 20:03:36.750948 26581 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [term 1 LEADER]: Becoming Leader. State: Replica: 0d5c35c3ac9e4692a519eb6b7a543e74, State: Running, Role: LEADER
I20250901 20:03:36.751307 26581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5c35c3ac9e4692a519eb6b7a543e74" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46107 } }
I20250901 20:03:36.751681 26576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 20:03:36.752521 26581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d5c35c3ac9e4692a519eb6b7a543e74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5c35c3ac9e4692a519eb6b7a543e74" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46107 } } }
I20250901 20:03:36.752730 26581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [sys.catalog]: This master's current role is: LEADER
I20250901 20:03:36.753055 26581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d5c35c3ac9e4692a519eb6b7a543e74. Latest consensus state: current_term: 1 leader_uuid: "0d5c35c3ac9e4692a519eb6b7a543e74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5c35c3ac9e4692a519eb6b7a543e74" member_type: VOTER last_known_addr { host: "127.14.67.126" port: 46107 } } }
I20250901 20:03:36.753198 26581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74 [sys.catalog]: This master's current role is: LEADER
I20250901 20:03:36.753269 26579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:36.739671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.14.67.126:46107)
0901 20:03:36.740107 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0901 20:03:36.740120 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0901 20:03:36.740163 (+ 43us) server_negotiation.cc:407] Connection header received
0901 20:03:36.740869 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:03:36.740891 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:03:36.741254 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:03:36.741598 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:03:36.742659 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:03:36.743872 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:03:36.744691 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:03:36.744966 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:03:36.747690 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:03:36.747707 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:03:36.747728 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:03:36.747762 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:03:36.749933 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:03:36.750549 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:03:36.750557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:03:36.750562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:03:36.750636 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:03:36.750930 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:03:36.750932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:03:36.750934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:03:36.751312 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:03:36.752126 (+ 814us) server_negotiation.cc:1035] Waiting for connection context
0901 20:03:36.752284 (+ 158us) server_negotiation.cc:299] Negotiation successful
0901 20:03:36.752504 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":128,"threads_started":1}
I20250901 20:03:36.753957 26576 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20250901 20:03:36.753528 26590 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 20:03:36.761976 26590 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 20:03:36.767644 26590 catalog_manager.cc:1349] Generated new cluster ID: d242b1e2d7e548a0a1cb30c7574b0df5
I20250901 20:03:36.767699 26590 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 20:03:36.776932 26590 catalog_manager.cc:1372] Generated new certificate authority record
I20250901 20:03:36.778111 26590 catalog_manager.cc:1506] Loading token signing keys...
I20250901 20:03:36.800436 26590 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0d5c35c3ac9e4692a519eb6b7a543e74: Generated new TSK 0
I20250901 20:03:36.801291 26590 catalog_manager.cc:1516] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.14.67.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:03:37.538686 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.67.84:36277/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.65:0
--local_ip_for_outbound_sockets=127.14.67.65
--webserver_interface=127.14.67.65
--webserver_port=0
--tserver_master_addrs=127.14.67.126:46107
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:38159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20250901 20:03:37.653353 26623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:03:37.653546 26623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:03:37.653635 26623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:03:37.656783 26623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:03:37.656844 26623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:03:37.656952 26623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.65
I20250901 20:03:37.660843 26623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:38159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.67.84:36277/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.67.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:46107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.65
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:03:37.661790 26623 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:03:37.662599 26623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:03:37.668371 26633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:03:37.668370 26631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:03:37.668844 26623 server_base.cc:1047] running on GCE node
W20250901 20:03:37.668884 26630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:03:37.669158 26623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:03:37.669610 26623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:03:37.670732 26623 hybrid_clock.cc:648] HybridClock initialized: now 1756757017670721 us; error 25 us; skew 500 ppm
Sep 01 20:03:37 dist-test-slave-2s4d krb5kdc[26037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757017, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:03:37.673266 26623 init.cc:377] Logged in from keytab as kudu/127.14.67.65@KRBTEST.COM (short username kudu)
I20250901 20:03:37.674238 26623 webserver.cc:480] Webserver started at http://127.14.67.65:35763/ using document root <none> and password file <none>
I20250901 20:03:37.674782 26623 fs_manager.cc:362] Metadata directory not provided
I20250901 20:03:37.674835 26623 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:03:37.674997 26623 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 01 20:03:37 dist-test-slave-2s4d krb5kdc[26037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757017, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for HTTP/127.14.67.84@KRBTEST.COM
I20250901 20:03:37.693822 26623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/data/instance:
uuid: "f37d26b004a24529958d11191be49d84"
format_stamp: "Formatted at 2025-09-01 20:03:37 on dist-test-slave-2s4d"
server_key: "5eab535ce7091c415302fe263655cb59"
server_key_iv: "01002d4015d04acd722d6b393a0b0220"
server_key_version: "kuduclusterkey@0"
I20250901 20:03:37.694478 26623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance:
uuid: "f37d26b004a24529958d11191be49d84"
format_stamp: "Formatted at 2025-09-01 20:03:37 on dist-test-slave-2s4d"
server_key: "5eab535ce7091c415302fe263655cb59"
server_key_iv: "01002d4015d04acd722d6b393a0b0220"
server_key_version: "kuduclusterkey@0"
I20250901 20:03:37.702886 26623 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.000s
I20250901 20:03:37.722926 26640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:03:37.724299 26623 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20250901 20:03:37.724493 26623 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/wal
uuid: "f37d26b004a24529958d11191be49d84"
format_stamp: "Formatted at 2025-09-01 20:03:37 on dist-test-slave-2s4d"
server_key: "5eab535ce7091c415302fe263655cb59"
server_key_iv: "01002d4015d04acd722d6b393a0b0220"
server_key_version: "kuduclusterkey@0"
I20250901 20:03:37.724687 26623 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:03:37.754702 26623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:03:37.758095 26623 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:03:37.758345 26623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:03:37.759027 26623 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:03:37.760088 26623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:03:37.760202 26623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:03:37.760327 26623 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:03:37.760411 26623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:03:37.772825 26623 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.65:43489
I20250901 20:03:37.773109 26755 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.65:43489 every 8 connection(s)
I20250901 20:03:37.773892 26623 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/data/info.pb
I20250901 20:03:37.779430 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 26623
I20250901 20:03:37.779559 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-0/wal/instance
I20250901 20:03:37.779827 14605 external_mini_cluster.cc:1468] Setting key 74817976cd23366b7928d40c1c7fe173
Sep 01 20:03:37 dist-test-slave-2s4d krb5kdc[26037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757017, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.65@KRBTEST.COM for kudu/127.14.67.126@KRBTEST.COM
I20250901 20:03:37.789000 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:37.776375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.65:37099)
0901 20:03:37.776798 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:37.776840 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0901 20:03:37.777624 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:03:37.778115 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0901 20:03:37.778122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0901 20:03:37.778359 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0901 20:03:37.778954 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:03:37.778966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:03:37.780031 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0901 20:03:37.780035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0901 20:03:37.780990 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0901 20:03:37.781002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0901 20:03:37.781530 (+ 528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:03:37.782214 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0901 20:03:37.782250 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0901 20:03:37.783655 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0901 20:03:37.786085 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:03:37.786089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:03:37.786101 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:03:37.786339 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:03:37.786648 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0901 20:03:37.786653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0901 20:03:37.786658 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0901 20:03:37.786787 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0901 20:03:37.787249 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0901 20:03:37.787258 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0901 20:03:37.787591 (+ 333us) client_negotiation.cc:769] Sending connection context
0901 20:03:37.787838 (+ 247us) client_negotiation.cc:240] Negotiation successful
0901 20:03:37.788112 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":101,"threads_started":1}
I20250901 20:03:37.790071 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:37.776131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.67.65:37099 (local address 127.14.67.126:46107)
0901 20:03:37.776443 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0901 20:03:37.776448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0901 20:03:37.776925 (+ 477us) server_negotiation.cc:407] Connection header received
0901 20:03:37.777812 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0901 20:03:37.777815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0901 20:03:37.777876 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0901 20:03:37.777961 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0901 20:03:37.779101 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:03:37.779881 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0901 20:03:37.781116 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0901 20:03:37.781308 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0901 20:03:37.783794 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0901 20:03:37.783817 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0901 20:03:37.783823 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0901 20:03:37.783854 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0901 20:03:37.785977 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:03:37.786440 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:03:37.786445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:03:37.786448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:03:37.786531 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0901 20:03:37.786896 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0901 20:03:37.786901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0901 20:03:37.786906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0901 20:03:37.787093 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0901 20:03:37.787192 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0901 20:03:37.789750 (+ 2558us) server_negotiation.cc:299] Negotiation successful
0901 20:03:37.789897 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":113,"threads_started":1}
I20250901 20:03:37.791079 26756 heartbeater.cc:344] Connected to a master server at 127.14.67.126:46107
I20250901 20:03:37.791314 26756 heartbeater.cc:461] Registering TS with master...
I20250901 20:03:37.791926 26756 heartbeater.cc:507] Master 127.14.67.126:46107 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.67.66@KRBTEST.COM; defaulting to no policy
I20250901 20:03:37.793437 26540 ts_manager.cc:194] Registered new tserver with Master: f37d26b004a24529958d11191be49d84 (127.14.67.65:43489)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.67.66@KRBTEST.COM" created.
I20250901 20:03:37.794584 26540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.67.65@KRBTEST.COM'} at 127.14.67.65:37099
Exception in thread "main" Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab.
org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.14.67.126 from keytab /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
WARNING: no policy specified for HTTP/127.14.67.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.67.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.14.67.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab.
I20250901 20:03:37.829480 14605 external_mini_cluster.cc:1366] Running /tmp/dist-test-task6kzUjy/build/debug/bin/kudu
/tmp/dist-test-task6kzUjy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.67.84:36277/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.67.66:0
--local_ip_for_outbound_sockets=127.14.67.66
--webserver_interface=127.14.67.66
--webserver_port=0
--tserver_master_addrs=127.14.67.126:46107
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.14.67.84:38159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20250901 20:03:37.830147 26615 server.cc:273] Received an EOF from the subprocess
F20250901 20:03:37.830825 26600 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f3f8882adcd google::LogMessage::Fail() at ??:0
@ 0x7f3f8882eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3f8882a7cc google::LogMessage::Flush() at ??:0
@ 0x7f3f8882bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3f8a4312d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f3f8a42e0d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f3f8a432005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x564d45ac0364 std::function<>::operator()() at ??:0
@ 0x7f3f89c7c60a kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3f8a46d6db start_thread at ??:0
@ 0x7f3f879e171f clone at ??:0
W20250901 20:03:37.980901 26765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 20:03:37.981153 26765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 20:03:37.981235 26765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250901 20:03:37.985678 26765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20250901 20:03:37.985750 26765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 20:03:37.985859 26765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.67.66
I20250901 20:03:37.991689 26765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.67.84:38159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.67.84:36277/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/krb5kdc/kudu.keytab
--principal=kudu/127.14.67.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.67.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.67.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.14.67.126:46107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.67.66
--log_dir=/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type DEBUG
built by None at 01 Sep 2025 19:43:16 UTC on 24a791456cd2
build id 7850
I20250901 20:03:37.992913 26765 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 20:03:37.993906 26765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250901 20:03:38.001125 26782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:03:38.001163 26784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 20:03:38.001132 26781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 20:03:38.001567 26765 server_base.cc:1047] running on GCE node
I20250901 20:03:38.001885 26765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 20:03:38.002355 26765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 20:03:38.003504 26765 hybrid_clock.cc:648] HybridClock initialized: now 1756757018003501 us; error 42 us; skew 500 ppm
Sep 01 20:03:38 dist-test-slave-2s4d krb5kdc[26037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757018, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20250901 20:03:38.005894 26765 init.cc:377] Logged in from keytab as kudu/127.14.67.66@KRBTEST.COM (short username kudu)
I20250901 20:03:38.006860 26765 webserver.cc:480] Webserver started at http://127.14.67.66:43745/ using document root <none> and password file <none>
I20250901 20:03:38.007362 26765 fs_manager.cc:362] Metadata directory not provided
I20250901 20:03:38.007411 26765 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 20:03:38.007583 26765 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Sep 01 20:03:38 dist-test-slave-2s4d krb5kdc[26037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1756757018, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.67.66@KRBTEST.COM for HTTP/127.14.67.84@KRBTEST.COM
I20250901 20:03:38.021924 26765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/data/instance:
uuid: "3d0b610c03664b2ba414be1e0832fea7"
format_stamp: "Formatted at 2025-09-01 20:03:38 on dist-test-slave-2s4d"
server_key: "74a67a756c903c45771de526a1acfc72"
server_key_iv: "8bd89acf7cf3d5374e47a6e8d28514f5"
server_key_version: "kuduclusterkey@0"
I20250901 20:03:38.022850 26765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance:
uuid: "3d0b610c03664b2ba414be1e0832fea7"
format_stamp: "Formatted at 2025-09-01 20:03:38 on dist-test-slave-2s4d"
server_key: "74a67a756c903c45771de526a1acfc72"
server_key_iv: "8bd89acf7cf3d5374e47a6e8d28514f5"
server_key_version: "kuduclusterkey@0"
I20250901 20:03:38.027374 26765 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250901 20:03:38.041195 26791 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 20:03:38.042409 26765 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250901 20:03:38.042553 26765 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/data,/tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/wal
uuid: "3d0b610c03664b2ba414be1e0832fea7"
format_stamp: "Formatted at 2025-09-01 20:03:38 on dist-test-slave-2s4d"
server_key: "74a67a756c903c45771de526a1acfc72"
server_key_iv: "8bd89acf7cf3d5374e47a6e8d28514f5"
server_key_version: "kuduclusterkey@0"
I20250901 20:03:38.042661 26765 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 20:03:38.056560 26765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 20:03:38.059280 26765 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 20:03:38.059459 26765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 20:03:38.059976 26765 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250901 20:03:38.060806 26765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250901 20:03:38.060858 26765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250901 20:03:38.060912 26765 ts_tablet_manager.cc:610] Registered 0 tablets
I20250901 20:03:38.060947 26765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250901 20:03:38.071830 26765 rpc_server.cc:307] RPC server started. Bound to: 127.14.67.66:33015
I20250901 20:03:38.072101 26904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.67.66:33015 every 8 connection(s)
I20250901 20:03:38.072882 26765 server_base.cc:1179] Dumped server information to /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/data/info.pb
W20250901 20:03:38.077913 26907 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 20:03:38.074839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.66:45717)
0901 20:03:38.075274 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:38.075317 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0901 20:03:38.076036 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 20:03:38.077249 (+ 1213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
W20250901 20:03:38.078295 26905 heartbeater.cc:646] Failed to heartbeat to 127.14.67.126:46107 (0 consecutive failures): Network error: Failed to ping master at 127.14.67.126:46107: Client connection negotiation failed: client connection to 127.14.67.126:46107: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 20:03:38.078913 26907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:38.078667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.66:41123)
0901 20:03:38.078789 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:38.078848 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20250901 20:03:38.079491 26907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:38.079288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.66:35563)
0901 20:03:38.079389 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:38.079434 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20250901 20:03:38.079672 26905 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250901 20:03:38.082621 14605 external_mini_cluster.cc:1428] Started /tmp/dist-test-task6kzUjy/build/debug/bin/kudu as pid 26765
I20250901 20:03:38.082748 14605 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task6kzUjy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1756756873705473-14605-0/minicluster-data/ts-1/wal/instance
I20250901 20:03:38.082971 14605 external_mini_cluster.cc:1468] Setting key 5e8c505f46ba166f5d37cf0c8b86d658
I20250901 20:03:38.797932 26756 heartbeater.cc:499] Master 127.14.67.126:46107 was elected leader, sending a full tablet report...
I20250901 20:03:38.798803 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:38.798389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.65:48083)
0901 20:03:38.798653 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:38.798713 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":69,"threads_started":1}
W20250901 20:03:38.799054 26756 heartbeater.cc:646] Failed to heartbeat to 127.14.67.126:46107 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
I20250901 20:03:39.080703 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:39.080268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.66:58883)
0901 20:03:39.080544 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:39.080601 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":75,"threads_started":1}
I20250901 20:03:39.800009 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:39.799627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.65:43739)
0901 20:03:39.799867 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:39.799920 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20250901 20:03:40.081853 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:40.081452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.66:55091)
0901 20:03:40.081712 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:40.081768 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20250901 20:03:40.801223 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:40.800807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.65:60391)
0901 20:03:40.801065 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:40.801123 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20250901 20:03:41.082955 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:41.082592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.66:44517)
0901 20:03:41.082820 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:41.082870 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20250901 20:03:41.802552 26915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:41.802064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.65:35865)
0901 20:03:41.802363 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:41.802423 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
I20250901 20:03:42.084034 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:42.083624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.66:45585)
0901 20:03:42.083882 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:42.083942 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20250901 20:03:42.803898 26917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:42.803365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.65:37965)
0901 20:03:42.803745 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:42.803804 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":77,"threads_started":1}
I20250901 20:03:43.085441 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0901 20:03:43.084924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.67.126:46107 (local address 127.14.67.66:43273)
0901 20:03:43.085188 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0901 20:03:43.085244 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.67.126:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
F20250901 20:03:43.109475 14605 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1756757023 (unix time) try "date -d @1756757023" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000390d) received by PID 14605 (TID 0x7fed9af61d80) from PID 14605; stack trace: ***
@ 0x7feda6a7e980 (unknown) at ??:0
@ 0x7feda4165fb7 gsignal at ??:0
@ 0x7feda4167921 abort at ??:0
@ 0x7feda53d3dcd google::LogMessage::Fail() at ??:0
@ 0x7feda53d7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7feda53d37cc google::LogMessage::Flush() at ??:0
@ 0x7feda53d4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7feda804fc29 kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7feda804c946 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x55c474b9d919 kudu::SecurityITest::StartCluster() at ??:0
@ 0x55c474b85792 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7feda63d90ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7feda63cdbdb testing::Test::Run() at ??:0
@ 0x7feda63cdd9d testing::TestInfo::Run() at ??:0
@ 0x7feda63ce377 testing::TestSuite::Run() at ??:0
@ 0x7feda63ce77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7feda63d960d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7feda63cde63 testing::UnitTest::Run() at ??:0
@ 0x55c474bd5c46 RUN_ALL_TESTS() at ??:0
@ 0x55c474bd4e33 main at ??:0
@ 0x7feda4148bf7 __libc_start_main at ??:0
@ 0x55c474b6d06a _start at ??:0
2025-09-01 20:03:43.306 UTC [26380] FATAL: terminating connection due to unexpected postmaster exit
2025-09-01 20:03:43.306 UTC [26383] FATAL: terminating connection due to unexpected postmaster exit
2025-09-01 20:03:43.306 UTC [26382] FATAL: terminating connection due to unexpected postmaster exit
2025-09-01 20:03:43.307 UTC [26486] FATAL: terminating connection due to unexpected postmaster exit
2025-09-01 20:03:43.307 UTC [26378] FATAL: terminating connection due to unexpected postmaster exit
2025-09-01 20:03:43.309 UTC [26379] FATAL: terminating connection due to unexpected postmaster exit